2024-09-06T14:11:11.501 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-06T14:11:11.517 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-06T14:11:11.616 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/adking-2024-09-06_13:43:09-orch:cephadm-wip-athakkar-testing-2024-09-04-1837-distro-default-smithi/7892315 branch: wip-athakkar-testing-2024-09-04-1837 description: orch:cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rados_api_tests} email: adking@redhat.com first_in_suite: false job_id: '7892315' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: adking-2024-09-06_13:43:09-orch:cephadm-wip-athakkar-testing-2024-09-04-1837-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-athakkar-testing-2024-09-04-1837 ceph: conf: client: debug ms: 1 global: mon election default strategy: 1 ms bind msgr1: false ms bind msgr2: true ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon warn on pool no app: false osd: debug ms: 1 debug osd: 20 osd class default list: '*' osd class load list: '*' osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - reached quota - but it is still running - overall HEALTH_ - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(CACHE_POOL_NO_HIT_SET\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(PG_DEGRADED\) - CEPHADM_STRAY_DAEMON log-only-match: - CEPHADM_ sha1: 8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} cephadm: cephadm_mode: cephadm-package install: ceph: flavor: default sha1: 8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 extra_packages: - cephadm selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 workunit: branch: wip-athakkar-testing-2024-09-04-1837 sha1: 8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 owner: scheduled_adking@teuthology priority: 80 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - ceph.rgw.foo.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b - ceph.iscsi.iscsi.a seed: 9103 sha1: 8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 sleep_before_teardown: 0 subset: 1/20 suite: orch:cephadm suite_branch: wip-athakkar-testing-2024-09-04-1837 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_8b38c33cecf4b0b7a9b3116eb9089e21e444ac73/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 targets: smithi142.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLdY989DHPRQ8BRk7Ro+x0IJ0Z91Eb5p9u88R130M9UoX7X8HIYkfFy0SWaoNV/zFdKWWAyVTe+mQSmDIjIUmAI= smithi162.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP6Q/r9umtmS7whkVQBxBRYe1CIdsk+1RkljMsWzEGxBNm7zXHo0A0vFmbfffWi6P1zVKiDT1zJMzVd5rPshxLo= tasks: - pexec: all: - sudo dnf remove nvme-cli -y - sudo dnf install nvmetcli nvme-cli -y - install: null - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - workunit: clients: client.0: - rados/test.sh - rados/test_pool_quota.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-06_13:43:09 tube: smithi user: adking verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-06T14:11:11.616 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_8b38c33cecf4b0b7a9b3116eb9089e21e444ac73/qa; will attempt to use it 2024-09-06T14:11:11.617 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_8b38c33cecf4b0b7a9b3116eb9089e21e444ac73/qa/tasks 2024-09-06T14:11:11.617 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-06T14:11:11.619 INFO:teuthology.task.internal:Checking packages... 2024-09-06T14:11:11.641 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '8b38c33cecf4b0b7a9b3116eb9089e21e444ac73' 2024-09-06T14:11:11.641 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-06T14:11:11.642 INFO:teuthology.packaging:ref: None 2024-09-06T14:11:11.642 INFO:teuthology.packaging:tag: None 2024-09-06T14:11:11.642 INFO:teuthology.packaging:branch: wip-athakkar-testing-2024-09-04-1837 2024-09-06T14:11:11.642 INFO:teuthology.packaging:sha1: 8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 2024-09-06T14:11:11.642 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-athakkar-testing-2024-09-04-1837 2024-09-06T14:11:11.799 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-4704.g8b38c33c 2024-09-06T14:11:11.800 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-06T14:11:11.807 INFO:teuthology.task.internal:no buildpackages task found 2024-09-06T14:11:11.807 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-06T14:11:11.814 INFO:teuthology.task.internal:Saving configuration 2024-09-06T14:11:11.826 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-06T14:11:11.832 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-06T14:11:11.859 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi142.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/adking-2024-09-06_13:43:09-orch:cephadm-wip-athakkar-testing-2024-09-04-1837-distro-default-smithi/7892315', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-06 14:05:56.533422', 'locked_by': 'scheduled_adking@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLdY989DHPRQ8BRk7Ro+x0IJ0Z91Eb5p9u88R130M9UoX7X8HIYkfFy0SWaoNV/zFdKWWAyVTe+mQSmDIjIUmAI='} 2024-09-06T14:11:11.886 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi162.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/adking-2024-09-06_13:43:09-orch:cephadm-wip-athakkar-testing-2024-09-04-1837-distro-default-smithi/7892315', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-06 14:05:56.534837', 'locked_by': 'scheduled_adking@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP6Q/r9umtmS7whkVQBxBRYe1CIdsk+1RkljMsWzEGxBNm7zXHo0A0vFmbfffWi6P1zVKiDT1zJMzVd5rPshxLo='} 2024-09-06T14:11:11.886 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-06T14:11:11.892 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2024-09-06T14:11:11.892 INFO:teuthology.task.internal:roles: ubuntu@smithi162.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b', 'ceph.iscsi.iscsi.a'] 2024-09-06T14:11:11.892 INFO:teuthology.run_tasks:Running task console_log... 2024-09-06T14:11:11.968 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7efcc192fac0>, signals=[15]) 2024-09-06T14:11:11.968 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-06T14:11:11.974 INFO:teuthology.task.internal:Opening connections... 2024-09-06T14:11:11.974 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi142.front.sepia.ceph.com 2024-09-06T14:11:11.975 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi142.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-06T14:11:12.076 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi162.front.sepia.ceph.com 2024-09-06T14:11:12.077 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-06T14:11:12.158 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-06T14:11:12.165 DEBUG:teuthology.orchestra.run.smithi142:> uname -m 2024-09-06T14:11:12.182 INFO:teuthology.orchestra.run.smithi142.stdout:x86_64 2024-09-06T14:11:12.182 DEBUG:teuthology.orchestra.run.smithi142:> cat /etc/os-release 2024-09-06T14:11:12.239 INFO:teuthology.orchestra.run.smithi142.stdout:NAME="CentOS Stream" 2024-09-06T14:11:12.239 INFO:teuthology.orchestra.run.smithi142.stdout:VERSION="9" 2024-09-06T14:11:12.240 INFO:teuthology.orchestra.run.smithi142.stdout:ID="centos" 2024-09-06T14:11:12.240 INFO:teuthology.orchestra.run.smithi142.stdout:ID_LIKE="rhel fedora" 2024-09-06T14:11:12.240 INFO:teuthology.orchestra.run.smithi142.stdout:VERSION_ID="9" 2024-09-06T14:11:12.240 INFO:teuthology.orchestra.run.smithi142.stdout:PLATFORM_ID="platform:el9" 2024-09-06T14:11:12.240 INFO:teuthology.orchestra.run.smithi142.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-06T14:11:12.240 INFO:teuthology.orchestra.run.smithi142.stdout:ANSI_COLOR="0;31" 2024-09-06T14:11:12.240 INFO:teuthology.orchestra.run.smithi142.stdout:LOGO="fedora-logo-icon" 2024-09-06T14:11:12.240 INFO:teuthology.orchestra.run.smithi142.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-06T14:11:12.240 INFO:teuthology.orchestra.run.smithi142.stdout:HOME_URL="https://centos.org/" 2024-09-06T14:11:12.240 INFO:teuthology.orchestra.run.smithi142.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-06T14:11:12.240 INFO:teuthology.orchestra.run.smithi142.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-06T14:11:12.240 INFO:teuthology.orchestra.run.smithi142.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-06T14:11:12.241 INFO:teuthology.lock.ops:Updating smithi142.front.sepia.ceph.com on lock server 2024-09-06T14:11:12.277 DEBUG:teuthology.orchestra.run.smithi162:> uname -m 2024-09-06T14:11:12.295 INFO:teuthology.orchestra.run.smithi162.stdout:x86_64 2024-09-06T14:11:12.295 DEBUG:teuthology.orchestra.run.smithi162:> cat /etc/os-release 2024-09-06T14:11:12.354 INFO:teuthology.orchestra.run.smithi162.stdout:NAME="CentOS Stream" 2024-09-06T14:11:12.354 INFO:teuthology.orchestra.run.smithi162.stdout:VERSION="9" 2024-09-06T14:11:12.354 INFO:teuthology.orchestra.run.smithi162.stdout:ID="centos" 2024-09-06T14:11:12.354 INFO:teuthology.orchestra.run.smithi162.stdout:ID_LIKE="rhel fedora" 2024-09-06T14:11:12.354 INFO:teuthology.orchestra.run.smithi162.stdout:VERSION_ID="9" 2024-09-06T14:11:12.354 INFO:teuthology.orchestra.run.smithi162.stdout:PLATFORM_ID="platform:el9" 2024-09-06T14:11:12.354 INFO:teuthology.orchestra.run.smithi162.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-06T14:11:12.354 INFO:teuthology.orchestra.run.smithi162.stdout:ANSI_COLOR="0;31" 2024-09-06T14:11:12.354 INFO:teuthology.orchestra.run.smithi162.stdout:LOGO="fedora-logo-icon" 2024-09-06T14:11:12.354 INFO:teuthology.orchestra.run.smithi162.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-06T14:11:12.355 INFO:teuthology.orchestra.run.smithi162.stdout:HOME_URL="https://centos.org/" 2024-09-06T14:11:12.355 INFO:teuthology.orchestra.run.smithi162.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-06T14:11:12.355 INFO:teuthology.orchestra.run.smithi162.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-06T14:11:12.355 INFO:teuthology.orchestra.run.smithi162.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-06T14:11:12.355 INFO:teuthology.lock.ops:Updating smithi162.front.sepia.ceph.com on lock server 2024-09-06T14:11:12.385 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-06T14:11:12.395 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-06T14:11:12.401 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-06T14:11:12.401 DEBUG:teuthology.orchestra.run.smithi142:> test '!' -e /home/ubuntu/cephtest 2024-09-06T14:11:12.404 DEBUG:teuthology.orchestra.run.smithi162:> test '!' -e /home/ubuntu/cephtest 2024-09-06T14:11:12.420 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-06T14:11:12.426 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-06T14:11:12.426 DEBUG:teuthology.orchestra.run.smithi142:> test -z $(ls -A /var/lib/ceph) 2024-09-06T14:11:12.461 DEBUG:teuthology.orchestra.run.smithi162:> test -z $(ls -A /var/lib/ceph) 2024-09-06T14:11:12.560 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-06T14:11:12.624 INFO:teuthology.run_tasks:Running task kernel... 2024-09-06T14:11:12.639 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-06T14:11:12.640 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-06T14:11:12.640 DEBUG:teuthology.orchestra.run.smithi142:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-06T14:11:12.640 DEBUG:teuthology.orchestra.run.smithi162:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-06T14:11:12.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-06T14:11:12.657 DEBUG:teuthology.orchestra.run.smithi142:> uname -r 2024-09-06T14:11:12.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-06T14:11:12.659 DEBUG:teuthology.orchestra.run.smithi162:> uname -r 2024-09-06T14:11:12.715 INFO:teuthology.orchestra.run.smithi142.stdout:5.14.0-503.el9.x86_64 2024-09-06T14:11:12.715 INFO:teuthology.task.kernel:Running kernel on smithi142: 5.14.0-503.el9.x86_64 2024-09-06T14:11:12.715 DEBUG:teuthology.orchestra.run.smithi142:> sudo yum install -y kernel 2024-09-06T14:11:12.716 INFO:teuthology.orchestra.run.smithi162.stdout:5.14.0-503.el9.x86_64 2024-09-06T14:11:12.717 INFO:teuthology.task.kernel:Running kernel on smithi162: 5.14.0-503.el9.x86_64 2024-09-06T14:11:12.717 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum install -y kernel 2024-09-06T14:11:16.251 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - BaseOS 4.7 MB/s | 8.2 MB 00:01 2024-09-06T14:11:18.360 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - AppStream 20 MB/s | 20 MB 00:01 2024-09-06T14:11:18.990 INFO:teuthology.orchestra.run.smithi142.stdout:CentOS Stream 9 - BaseOS 1.8 MB/s | 8.2 MB 00:04 2024-09-06T14:11:23.341 INFO:teuthology.orchestra.run.smithi142.stdout:CentOS Stream 9 - AppStream 6.3 MB/s | 20 MB 00:03 2024-09-06T14:11:24.043 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - CRB 3.5 MB/s | 6.5 MB 00:01 2024-09-06T14:11:25.844 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - Extras packages 60 kB/s | 19 kB 00:00 2024-09-06T14:11:27.397 INFO:teuthology.orchestra.run.smithi162.stdout:Extra Packages for Enterprise Linux 16 MB/s | 23 MB 00:01 2024-09-06T14:11:28.133 INFO:teuthology.orchestra.run.smithi142.stdout:CentOS Stream 9 - CRB 6.7 MB/s | 6.5 MB 00:00 2024-09-06T14:11:29.583 INFO:teuthology.orchestra.run.smithi142.stdout:CentOS Stream 9 - Extras packages 76 kB/s | 19 kB 00:00 2024-09-06T14:11:30.746 INFO:teuthology.orchestra.run.smithi142.stdout:Extra Packages for Enterprise Linux 22 MB/s | 23 MB 00:01 2024-09-06T14:11:32.811 INFO:teuthology.orchestra.run.smithi162.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-09-06T14:11:34.843 INFO:teuthology.orchestra.run.smithi162.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-06T14:11:34.843 INFO:teuthology.orchestra.run.smithi162.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-06T14:11:34.921 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-09-06T14:11:34.921 INFO:teuthology.orchestra.run.smithi162.stdout:Nothing to do. 2024-09-06T14:11:34.921 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-09-06T14:11:35.011 DEBUG:teuthology.orchestra.run.smithi162:> echo no | sudo yum reinstall kernel || true 2024-09-06T14:11:35.561 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:00:03 ago on Fri 06 Sep 2024 02:11:32 PM UTC. 2024-09-06T14:11:35.694 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-09-06T14:11:35.698 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-06T14:11:35.698 INFO:teuthology.orchestra.run.smithi162.stdout: Package Architecture Version Repository Size 2024-09-06T14:11:35.699 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-06T14:11:35.699 INFO:teuthology.orchestra.run.smithi162.stdout:Reinstalling: 2024-09-06T14:11:35.699 INFO:teuthology.orchestra.run.smithi162.stdout: kernel x86_64 5.14.0-503.el9 baseos 1.9 M 2024-09-06T14:11:35.699 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-06T14:11:35.699 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-09-06T14:11:35.699 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-06T14:11:35.699 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-06T14:11:35.699 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 1.9 M 2024-09-06T14:11:35.699 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 0 2024-09-06T14:11:35.699 INFO:teuthology.orchestra.run.smithi162.stderr:Operation aborted. 2024-09-06T14:11:35.757 INFO:teuthology.orchestra.run.smithi162.stdout:Is this ok [y/N]: 2024-09-06T14:11:35.758 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum reinstall -y kernel || true 2024-09-06T14:11:36.251 INFO:teuthology.orchestra.run.smithi142.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-09-06T14:11:36.300 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:00:04 ago on Fri 06 Sep 2024 02:11:32 PM UTC. 2024-09-06T14:11:36.427 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-09-06T14:11:36.428 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-06T14:11:36.428 INFO:teuthology.orchestra.run.smithi162.stdout: Package Architecture Version Repository Size 2024-09-06T14:11:36.429 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-06T14:11:36.429 INFO:teuthology.orchestra.run.smithi162.stdout:Reinstalling: 2024-09-06T14:11:36.429 INFO:teuthology.orchestra.run.smithi162.stdout: kernel x86_64 5.14.0-503.el9 baseos 1.9 M 2024-09-06T14:11:36.429 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-06T14:11:36.429 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-09-06T14:11:36.429 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-06T14:11:36.429 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-06T14:11:36.429 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 1.9 M 2024-09-06T14:11:36.429 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 0 2024-09-06T14:11:36.429 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-09-06T14:11:37.621 INFO:teuthology.orchestra.run.smithi162.stdout:kernel-5.14.0-503.el9.x86_64.rpm 1.7 MB/s | 1.9 MB 00:01 2024-09-06T14:11:37.621 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-09-06T14:11:37.622 INFO:teuthology.orchestra.run.smithi162.stdout:Total 1.6 MB/s | 1.9 MB 00:01 2024-09-06T14:11:37.652 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-09-06T14:11:37.899 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-09-06T14:11:37.899 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-09-06T14:11:37.930 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-09-06T14:11:37.931 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-09-06T14:11:38.111 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-09-06T14:11:38.206 INFO:teuthology.orchestra.run.smithi162.stdout: Reinstalling : kernel-5.14.0-503.el9.x86_64 1/2 2024-09-06T14:11:38.207 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : kernel-5.14.0-503.el9.x86_64 2/2 2024-09-06T14:11:38.283 INFO:teuthology.orchestra.run.smithi142.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-06T14:11:38.283 INFO:teuthology.orchestra.run.smithi142.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-06T14:11:38.366 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2024-09-06T14:11:38.367 INFO:teuthology.orchestra.run.smithi142.stdout:Nothing to do. 2024-09-06T14:11:38.367 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2024-09-06T14:11:38.467 DEBUG:teuthology.orchestra.run.smithi142:> echo no | sudo yum reinstall kernel || true 2024-09-06T14:11:39.014 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: kernel-5.14.0-503.el9.x86_64 2/2 2024-09-06T14:11:39.014 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : kernel-5.14.0-503.el9.x86_64 1/2 2024-09-06T14:11:39.055 INFO:teuthology.orchestra.run.smithi142.stdout:Last metadata expiration check: 0:00:03 ago on Fri 06 Sep 2024 02:11:36 PM UTC. 2024-09-06T14:11:39.203 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2024-09-06T14:11:39.207 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-09-06T14:11:39.207 INFO:teuthology.orchestra.run.smithi142.stdout: Package Architecture Version Repository Size 2024-09-06T14:11:39.207 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-09-06T14:11:39.208 INFO:teuthology.orchestra.run.smithi142.stdout:Reinstalling: 2024-09-06T14:11:39.208 INFO:teuthology.orchestra.run.smithi142.stdout: kernel x86_64 5.14.0-503.el9 baseos 1.9 M 2024-09-06T14:11:39.208 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:11:39.208 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction Summary 2024-09-06T14:11:39.208 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-09-06T14:11:39.208 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:11:39.208 INFO:teuthology.orchestra.run.smithi142.stdout:Total download size: 1.9 M 2024-09-06T14:11:39.208 INFO:teuthology.orchestra.run.smithi142.stdout:Installed size: 0 2024-09-06T14:11:39.208 INFO:teuthology.orchestra.run.smithi142.stderr:Operation aborted. 2024-09-06T14:11:39.262 INFO:teuthology.orchestra.run.smithi142.stdout:Is this ok [y/N]: 2024-09-06T14:11:39.262 DEBUG:teuthology.orchestra.run.smithi142:> sudo yum reinstall -y kernel || true 2024-09-06T14:11:39.729 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : kernel-5.14.0-503.el9.x86_64 2/2 2024-09-06T14:11:39.729 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-06T14:11:39.729 INFO:teuthology.orchestra.run.smithi162.stdout:Reinstalled: 2024-09-06T14:11:39.729 INFO:teuthology.orchestra.run.smithi162.stdout: kernel-5.14.0-503.el9.x86_64 2024-09-06T14:11:39.729 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-06T14:11:39.730 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-09-06T14:11:39.804 INFO:teuthology.orchestra.run.smithi142.stdout:Last metadata expiration check: 0:00:03 ago on Fri 06 Sep 2024 02:11:36 PM UTC. 2024-09-06T14:11:39.909 DEBUG:teuthology.orchestra.run.smithi162:> rpm -q kernel | sort -rV | head -n 1 2024-09-06T14:11:39.931 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2024-09-06T14:11:39.932 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-09-06T14:11:39.932 INFO:teuthology.orchestra.run.smithi142.stdout: Package Architecture Version Repository Size 2024-09-06T14:11:39.932 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-09-06T14:11:39.933 INFO:teuthology.orchestra.run.smithi142.stdout:Reinstalling: 2024-09-06T14:11:39.933 INFO:teuthology.orchestra.run.smithi142.stdout: kernel x86_64 5.14.0-503.el9 baseos 1.9 M 2024-09-06T14:11:39.933 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:11:39.933 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction Summary 2024-09-06T14:11:39.933 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-09-06T14:11:39.933 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:11:39.933 INFO:teuthology.orchestra.run.smithi142.stdout:Total download size: 1.9 M 2024-09-06T14:11:39.933 INFO:teuthology.orchestra.run.smithi142.stdout:Installed size: 0 2024-09-06T14:11:39.933 INFO:teuthology.orchestra.run.smithi142.stdout:Downloading Packages: 2024-09-06T14:11:39.967 INFO:teuthology.orchestra.run.smithi162.stdout:kernel-5.14.0-503.el9.x86_64 2024-09-06T14:11:39.967 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-503.el9.x86_64 2024-09-06T14:11:39.968 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-09-06T14:11:39.968 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-06T14:11:39.968 DEBUG:teuthology.orchestra.run.smithi162:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-06T14:11:40.019 INFO:teuthology.orchestra.run.smithi162.stdout:ttyS1 2024-09-06T14:11:40.040 DEBUG:teuthology.parallel:result is None 2024-09-06T14:11:40.662 INFO:teuthology.orchestra.run.smithi142.stdout:kernel-5.14.0-503.el9.x86_64.rpm 3.0 MB/s | 1.9 MB 00:00 2024-09-06T14:11:40.662 INFO:teuthology.orchestra.run.smithi142.stdout:-------------------------------------------------------------------------------- 2024-09-06T14:11:40.662 INFO:teuthology.orchestra.run.smithi142.stdout:Total 2.6 MB/s | 1.9 MB 00:00 2024-09-06T14:11:40.694 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction check 2024-09-06T14:11:40.947 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction check succeeded. 2024-09-06T14:11:40.947 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction test 2024-09-06T14:11:40.977 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction test succeeded. 2024-09-06T14:11:40.977 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction 2024-09-06T14:11:41.135 INFO:teuthology.orchestra.run.smithi142.stdout: Preparing : 1/1 2024-09-06T14:11:41.239 INFO:teuthology.orchestra.run.smithi142.stdout: Reinstalling : kernel-5.14.0-503.el9.x86_64 1/2 2024-09-06T14:11:41.239 INFO:teuthology.orchestra.run.smithi142.stdout: Cleanup : kernel-5.14.0-503.el9.x86_64 2/2 2024-09-06T14:11:41.928 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: kernel-5.14.0-503.el9.x86_64 2/2 2024-09-06T14:11:41.929 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : kernel-5.14.0-503.el9.x86_64 1/2 2024-09-06T14:11:42.642 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : kernel-5.14.0-503.el9.x86_64 2/2 2024-09-06T14:11:42.643 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:11:42.643 INFO:teuthology.orchestra.run.smithi142.stdout:Reinstalled: 2024-09-06T14:11:42.643 INFO:teuthology.orchestra.run.smithi142.stdout: kernel-5.14.0-503.el9.x86_64 2024-09-06T14:11:42.643 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:11:42.643 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2024-09-06T14:11:42.841 DEBUG:teuthology.orchestra.run.smithi142:> rpm -q kernel | sort -rV | head -n 1 2024-09-06T14:11:42.900 INFO:teuthology.orchestra.run.smithi142.stdout:kernel-5.14.0-503.el9.x86_64 2024-09-06T14:11:42.900 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-503.el9.x86_64 2024-09-06T14:11:42.900 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-09-06T14:11:42.900 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-06T14:11:42.900 DEBUG:teuthology.orchestra.run.smithi142:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-06T14:11:42.936 INFO:teuthology.orchestra.run.smithi142.stdout:ttyS1 2024-09-06T14:11:42.956 DEBUG:teuthology.parallel:result is None 2024-09-06T14:11:42.956 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-06T14:11:42.963 INFO:teuthology.task.internal:Creating test directory... 2024-09-06T14:11:42.963 DEBUG:teuthology.orchestra.run.smithi142:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-06T14:11:42.999 DEBUG:teuthology.orchestra.run.smithi162:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-06T14:11:43.018 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-06T14:11:43.026 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-06T14:11:43.033 INFO:teuthology.task.internal:Creating archive directory... 2024-09-06T14:11:43.034 DEBUG:teuthology.orchestra.run.smithi142:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-06T14:11:43.061 DEBUG:teuthology.orchestra.run.smithi162:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-06T14:11:43.168 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-06T14:11:43.290 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-06T14:11:43.290 DEBUG:teuthology.orchestra.run.smithi142:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-06T14:11:43.297 DEBUG:teuthology.orchestra.run.smithi162:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-06T14:11:43.329 INFO:teuthology.orchestra.run.smithi142.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-06T14:11:43.339 INFO:teuthology.orchestra.run.smithi142.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-06T14:11:43.374 INFO:teuthology.orchestra.run.smithi162.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-06T14:11:43.387 INFO:teuthology.orchestra.run.smithi162.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-06T14:11:43.389 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-06T14:11:43.528 INFO:teuthology.task.internal:Configuring sudo... 2024-09-06T14:11:43.528 DEBUG:teuthology.orchestra.run.smithi142:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-06T14:11:43.531 DEBUG:teuthology.orchestra.run.smithi162:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-06T14:11:43.561 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-06T14:11:43.569 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-06T14:11:43.569 DEBUG:teuthology.orchestra.run.smithi142:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-06T14:11:43.596 DEBUG:teuthology.orchestra.run.smithi162:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-06T14:11:43.619 DEBUG:teuthology.orchestra.run.smithi142:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-06T14:11:43.696 DEBUG:teuthology.orchestra.run.smithi142:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-06T14:11:43.737 DEBUG:teuthology.orchestra.run.smithi142:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-06T14:11:43.821 DEBUG:teuthology.orchestra.run.smithi142:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-06T14:11:43.859 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-09-06T14:11:43.859 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-06T14:11:43.923 DEBUG:teuthology.orchestra.run.smithi162:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-06T14:11:43.976 DEBUG:teuthology.orchestra.run.smithi162:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-06T14:11:44.019 DEBUG:teuthology.orchestra.run.smithi162:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-06T14:11:44.113 DEBUG:teuthology.orchestra.run.smithi162:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-06T14:11:44.149 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-09-06T14:11:44.150 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-06T14:11:44.217 DEBUG:teuthology.orchestra.run.smithi142:> sudo service rsyslog restart 2024-09-06T14:11:44.219 DEBUG:teuthology.orchestra.run.smithi162:> sudo service rsyslog restart 2024-09-06T14:11:44.253 INFO:teuthology.orchestra.run.smithi142.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-06T14:11:44.301 INFO:teuthology.orchestra.run.smithi162.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-06T14:11:44.750 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-06T14:11:44.757 INFO:teuthology.task.internal:Starting timer... 2024-09-06T14:11:44.758 INFO:teuthology.run_tasks:Running task pcp... 2024-09-06T14:11:44.771 INFO:teuthology.run_tasks:Running task selinux... 2024-09-06T14:11:44.779 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-09-06T14:11:44.779 DEBUG:teuthology.orchestra.run.smithi142:> sudo service auditd rotate 2024-09-06T14:11:44.828 INFO:teuthology.orchestra.run.smithi142.stdout:Rotating logs: 2024-09-06T14:11:44.829 DEBUG:teuthology.orchestra.run.smithi162:> sudo service auditd rotate 2024-09-06T14:11:44.881 INFO:teuthology.orchestra.run.smithi162.stdout:Rotating logs: 2024-09-06T14:11:44.882 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-06T14:11:44.883 DEBUG:teuthology.orchestra.run.smithi142:> /usr/sbin/getenforce 2024-09-06T14:11:44.910 INFO:teuthology.orchestra.run.smithi142.stdout:Permissive 2024-09-06T14:11:44.911 DEBUG:teuthology.orchestra.run.smithi162:> /usr/sbin/getenforce 2024-09-06T14:11:44.938 INFO:teuthology.orchestra.run.smithi162.stdout:Permissive 2024-09-06T14:11:44.938 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi142.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi162.front.sepia.ceph.com': 'permissive'} 2024-09-06T14:11:44.939 DEBUG:teuthology.orchestra.run.smithi142:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-06T14:11:44.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-06T14:11:44.977 DEBUG:teuthology.orchestra.run.smithi162:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-06T14:11:45.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-06T14:11:45.004 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-06T14:11:45.004 DEBUG:teuthology.orchestra.run.smithi142:> sudo /usr/sbin/setenforce permissive 2024-09-06T14:11:45.051 DEBUG:teuthology.orchestra.run.smithi162:> sudo /usr/sbin/setenforce permissive 2024-09-06T14:11:45.086 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-06T14:11:45.095 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-06T14:11:45.106 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-06T14:11:45.120 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi142.front.sepia.ceph.com,smithi162.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-06T14:15:26.401 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi142.front.sepia.ceph.com'), Remote(name='ubuntu@smithi162.front.sepia.ceph.com')] 2024-09-06T14:15:26.450 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com' 2024-09-06T14:15:26.452 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi142.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-06T14:15:26.538 DEBUG:teuthology.orchestra.run.smithi142:> true 2024-09-06T14:15:26.620 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi142.front.sepia.ceph.com' 2024-09-06T14:15:26.621 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-09-06T14:15:26.621 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-06T14:15:26.704 DEBUG:teuthology.orchestra.run.smithi162:> true 2024-09-06T14:15:26.785 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-09-06T14:15:26.786 INFO:teuthology.run_tasks:Running task clock... 2024-09-06T14:15:26.798 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-06T14:15:26.799 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-06T14:15:26.799 DEBUG:teuthology.orchestra.run.smithi142:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-06T14:15:26.802 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-06T14:15:26.802 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-06T14:15:26.829 INFO:teuthology.orchestra.run.smithi142.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-06T14:15:26.845 INFO:teuthology.orchestra.run.smithi142.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-06T14:15:26.857 INFO:teuthology.orchestra.run.smithi162.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-06T14:15:26.869 INFO:teuthology.orchestra.run.smithi142.stderr:sudo: ntpd: command not found 2024-09-06T14:15:26.872 INFO:teuthology.orchestra.run.smithi162.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-06T14:15:26.880 INFO:teuthology.orchestra.run.smithi142.stdout:506 Cannot talk to daemon 2024-09-06T14:15:26.895 INFO:teuthology.orchestra.run.smithi142.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-06T14:15:26.897 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: ntpd: command not found 2024-09-06T14:15:26.908 INFO:teuthology.orchestra.run.smithi142.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-06T14:15:26.909 INFO:teuthology.orchestra.run.smithi162.stdout:506 Cannot talk to daemon 2024-09-06T14:15:26.923 INFO:teuthology.orchestra.run.smithi162.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-06T14:15:26.937 INFO:teuthology.orchestra.run.smithi162.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-06T14:15:26.971 INFO:teuthology.orchestra.run.smithi142.stderr:bash: line 1: ntpq: command not found 2024-09-06T14:15:26.974 INFO:teuthology.orchestra.run.smithi142.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-06T14:15:26.974 INFO:teuthology.orchestra.run.smithi142.stdout:=============================================================================== 2024-09-06T14:15:26.975 INFO:teuthology.orchestra.run.smithi142.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-06T14:15:26.975 INFO:teuthology.orchestra.run.smithi142.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-06T14:15:26.975 INFO:teuthology.orchestra.run.smithi142.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-06T14:15:26.975 INFO:teuthology.orchestra.run.smithi142.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-06T14:15:26.992 INFO:teuthology.orchestra.run.smithi162.stderr:bash: line 1: ntpq: command not found 2024-09-06T14:15:26.995 INFO:teuthology.orchestra.run.smithi162.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-06T14:15:26.995 INFO:teuthology.orchestra.run.smithi162.stdout:=============================================================================== 2024-09-06T14:15:26.995 INFO:teuthology.orchestra.run.smithi162.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-06T14:15:26.995 INFO:teuthology.orchestra.run.smithi162.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-06T14:15:26.996 INFO:teuthology.orchestra.run.smithi162.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-06T14:15:26.996 INFO:teuthology.orchestra.run.smithi162.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-06T14:15:26.996 INFO:teuthology.run_tasks:Running task pexec... 2024-09-06T14:15:27.007 INFO:teuthology.task.pexec:Executing custom commands... 2024-09-06T14:15:27.007 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi142.front.sepia.ceph.com 2024-09-06T14:15:27.008 DEBUG:teuthology.orchestra.run.smithi142:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-09-06T14:15:27.008 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi162.front.sepia.ceph.com 2024-09-06T14:15:27.008 DEBUG:teuthology.orchestra.run.smithi162:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-09-06T14:15:27.356 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2024-09-06T14:15:27.357 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-09-06T14:15:27.357 INFO:teuthology.orchestra.run.smithi142.stdout: Package Architecture Version Repository Size 2024-09-06T14:15:27.357 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-09-06T14:15:27.357 INFO:teuthology.orchestra.run.smithi142.stdout:Removing: 2024-09-06T14:15:27.358 INFO:teuthology.orchestra.run.smithi142.stdout: nvme-cli x86_64 2.9.1-6.el9 @baseos 5.3 M 2024-09-06T14:15:27.358 INFO:teuthology.orchestra.run.smithi142.stdout:Removing unused dependencies: 2024-09-06T14:15:27.358 INFO:teuthology.orchestra.run.smithi142.stdout: libnvme x86_64 1.9-3.el9 @baseos 273 k 2024-09-06T14:15:27.358 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:15:27.358 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction Summary 2024-09-06T14:15:27.358 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-09-06T14:15:27.358 INFO:teuthology.orchestra.run.smithi142.stdout:Remove 2 Packages 2024-09-06T14:15:27.358 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:15:27.358 INFO:teuthology.orchestra.run.smithi142.stdout:Freed space: 5.5 M 2024-09-06T14:15:27.358 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction check 2024-09-06T14:15:27.362 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction check succeeded. 2024-09-06T14:15:27.362 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction test 2024-09-06T14:15:27.375 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-09-06T14:15:27.376 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-06T14:15:27.377 INFO:teuthology.orchestra.run.smithi162.stdout: Package Architecture Version Repository Size 2024-09-06T14:15:27.377 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-06T14:15:27.377 INFO:teuthology.orchestra.run.smithi162.stdout:Removing: 2024-09-06T14:15:27.377 INFO:teuthology.orchestra.run.smithi162.stdout: nvme-cli x86_64 2.9.1-6.el9 @baseos 5.3 M 2024-09-06T14:15:27.377 INFO:teuthology.orchestra.run.smithi162.stdout:Removing unused dependencies: 2024-09-06T14:15:27.377 INFO:teuthology.orchestra.run.smithi162.stdout: libnvme x86_64 1.9-3.el9 @baseos 273 k 2024-09-06T14:15:27.377 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-06T14:15:27.377 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-09-06T14:15:27.377 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-06T14:15:27.377 INFO:teuthology.orchestra.run.smithi162.stdout:Remove 2 Packages 2024-09-06T14:15:27.377 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-06T14:15:27.377 INFO:teuthology.orchestra.run.smithi162.stdout:Freed space: 5.5 M 2024-09-06T14:15:27.377 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-09-06T14:15:27.381 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-09-06T14:15:27.381 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-09-06T14:15:27.401 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction test succeeded. 2024-09-06T14:15:27.402 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction 2024-09-06T14:15:27.417 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-09-06T14:15:27.417 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-09-06T14:15:27.517 INFO:teuthology.orchestra.run.smithi142.stdout: Preparing : 1/1 2024-09-06T14:15:27.532 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-09-06T14:15:27.613 INFO:teuthology.orchestra.run.smithi162.stdout: Erasing : nvme-cli-2.9.1-6.el9.x86_64 1/2 2024-09-06T14:15:27.616 INFO:teuthology.orchestra.run.smithi142.stdout: Erasing : nvme-cli-2.9.1-6.el9.x86_64 1/2 2024-09-06T14:15:27.669 INFO:teuthology.orchestra.run.smithi162.stdout: Erasing : libnvme-1.9-3.el9.x86_64 2/2 2024-09-06T14:15:27.676 INFO:teuthology.orchestra.run.smithi142.stdout: Erasing : libnvme-1.9-3.el9.x86_64 2/2 2024-09-06T14:15:28.072 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: libnvme-1.9-3.el9.x86_64 2/2 2024-09-06T14:15:28.072 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/2 2024-09-06T14:15:28.221 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: libnvme-1.9-3.el9.x86_64 2/2 2024-09-06T14:15:28.221 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/2 2024-09-06T14:15:28.290 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/2 2024-09-06T14:15:28.290 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-06T14:15:28.290 INFO:teuthology.orchestra.run.smithi162.stdout:Removed: 2024-09-06T14:15:28.290 INFO:teuthology.orchestra.run.smithi162.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-06T14:15:28.290 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-06T14:15:28.290 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-09-06T14:15:28.454 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/2 2024-09-06T14:15:28.454 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:15:28.454 INFO:teuthology.orchestra.run.smithi142.stdout:Removed: 2024-09-06T14:15:28.454 INFO:teuthology.orchestra.run.smithi142.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-06T14:15:28.454 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:15:28.454 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2024-09-06T14:15:29.121 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:00:47 ago on Fri 06 Sep 2024 02:14:41 PM UTC. 2024-09-06T14:15:29.270 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-09-06T14:15:29.270 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-06T14:15:29.270 INFO:teuthology.orchestra.run.smithi162.stdout: Package Architecture Version Repository Size 2024-09-06T14:15:29.270 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-06T14:15:29.270 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-09-06T14:15:29.271 INFO:teuthology.orchestra.run.smithi162.stdout: nvme-cli x86_64 2.9.1-6.el9 baseos 906 k 2024-09-06T14:15:29.271 INFO:teuthology.orchestra.run.smithi162.stdout: nvmetcli noarch 0.7-3.el9 baseos 44 k 2024-09-06T14:15:29.271 INFO:teuthology.orchestra.run.smithi162.stdout:Installing dependencies: 2024-09-06T14:15:29.271 INFO:teuthology.orchestra.run.smithi162.stdout: libnvme x86_64 1.9-3.el9 baseos 106 k 2024-09-06T14:15:29.271 INFO:teuthology.orchestra.run.smithi162.stdout: python3-configshell noarch 1:1.1.30-1.el9 baseos 72 k 2024-09-06T14:15:29.271 INFO:teuthology.orchestra.run.smithi162.stdout: python3-kmod x86_64 0.9-32.el9 baseos 84 k 2024-09-06T14:15:29.271 INFO:teuthology.orchestra.run.smithi162.stdout: python3-urwid x86_64 2.1.2-4.el9 baseos 837 k 2024-09-06T14:15:29.271 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-06T14:15:29.271 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-09-06T14:15:29.271 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-06T14:15:29.272 INFO:teuthology.orchestra.run.smithi162.stdout:Install 6 Packages 2024-09-06T14:15:29.272 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-06T14:15:29.272 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 2.0 M 2024-09-06T14:15:29.272 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 9.2 M 2024-09-06T14:15:29.272 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-09-06T14:15:29.275 INFO:teuthology.orchestra.run.smithi142.stdout:Last metadata expiration check: 0:00:49 ago on Fri 06 Sep 2024 02:14:40 PM UTC. 2024-09-06T14:15:29.409 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2024-09-06T14:15:29.410 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-09-06T14:15:29.410 INFO:teuthology.orchestra.run.smithi142.stdout: Package Architecture Version Repository Size 2024-09-06T14:15:29.410 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-09-06T14:15:29.410 INFO:teuthology.orchestra.run.smithi142.stdout:Installing: 2024-09-06T14:15:29.410 INFO:teuthology.orchestra.run.smithi142.stdout: nvme-cli x86_64 2.9.1-6.el9 baseos 906 k 2024-09-06T14:15:29.410 INFO:teuthology.orchestra.run.smithi142.stdout: nvmetcli noarch 0.7-3.el9 baseos 44 k 2024-09-06T14:15:29.410 INFO:teuthology.orchestra.run.smithi142.stdout:Installing dependencies: 2024-09-06T14:15:29.410 INFO:teuthology.orchestra.run.smithi142.stdout: libnvme x86_64 1.9-3.el9 baseos 106 k 2024-09-06T14:15:29.410 INFO:teuthology.orchestra.run.smithi142.stdout: python3-configshell noarch 1:1.1.30-1.el9 baseos 72 k 2024-09-06T14:15:29.410 INFO:teuthology.orchestra.run.smithi142.stdout: python3-kmod x86_64 0.9-32.el9 baseos 84 k 2024-09-06T14:15:29.410 INFO:teuthology.orchestra.run.smithi142.stdout: python3-urwid x86_64 2.1.2-4.el9 baseos 837 k 2024-09-06T14:15:29.410 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:15:29.410 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction Summary 2024-09-06T14:15:29.411 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-09-06T14:15:29.411 INFO:teuthology.orchestra.run.smithi142.stdout:Install 6 Packages 2024-09-06T14:15:29.411 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:15:29.411 INFO:teuthology.orchestra.run.smithi142.stdout:Total download size: 2.0 M 2024-09-06T14:15:29.411 INFO:teuthology.orchestra.run.smithi142.stdout:Installed size: 9.2 M 2024-09-06T14:15:29.411 INFO:teuthology.orchestra.run.smithi142.stdout:Downloading Packages: 2024-09-06T14:15:29.427 INFO:teuthology.orchestra.run.smithi162.stdout:(1/6): nvmetcli-0.7-3.el9.noarch.rpm 419 kB/s | 44 kB 00:00 2024-09-06T14:15:29.494 INFO:teuthology.orchestra.run.smithi162.stdout:(2/6): libnvme-1.9-3.el9.x86_64.rpm 613 kB/s | 106 kB 00:00 2024-09-06T14:15:29.520 INFO:teuthology.orchestra.run.smithi162.stdout:(3/6): python3-configshell-1.1.30-1.el9.noarch. 779 kB/s | 72 kB 00:00 2024-09-06T14:15:29.553 INFO:teuthology.orchestra.run.smithi162.stdout:(4/6): python3-kmod-0.9-32.el9.x86_64.rpm 1.4 MB/s | 84 kB 00:00 2024-09-06T14:15:29.617 INFO:teuthology.orchestra.run.smithi142.stdout:(1/6): nvmetcli-0.7-3.el9.noarch.rpm 287 kB/s | 44 kB 00:00 2024-09-06T14:15:29.667 INFO:teuthology.orchestra.run.smithi142.stdout:(2/6): libnvme-1.9-3.el9.x86_64.rpm 517 kB/s | 106 kB 00:00 2024-09-06T14:15:29.701 INFO:teuthology.orchestra.run.smithi142.stdout:(3/6): python3-configshell-1.1.30-1.el9.noarch. 859 kB/s | 72 kB 00:00 2024-09-06T14:15:29.720 INFO:teuthology.orchestra.run.smithi162.stdout:(5/6): nvme-cli-2.9.1-6.el9.x86_64.rpm 2.2 MB/s | 906 kB 00:00 2024-09-06T14:15:29.751 INFO:teuthology.orchestra.run.smithi142.stdout:(4/6): python3-kmod-0.9-32.el9.x86_64.rpm 1.0 MB/s | 84 kB 00:00 2024-09-06T14:15:29.785 INFO:teuthology.orchestra.run.smithi142.stdout:(5/6): nvme-cli-2.9.1-6.el9.x86_64.rpm 2.7 MB/s | 906 kB 00:00 2024-09-06T14:15:29.796 INFO:teuthology.orchestra.run.smithi162.stdout:(6/6): python3-urwid-2.1.2-4.el9.x86_64.rpm 3.0 MB/s | 837 kB 00:00 2024-09-06T14:15:29.796 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-09-06T14:15:29.796 INFO:teuthology.orchestra.run.smithi162.stdout:Total 3.8 MB/s | 2.0 MB 00:00 2024-09-06T14:15:29.869 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-09-06T14:15:29.883 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-09-06T14:15:29.883 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-09-06T14:15:29.894 INFO:teuthology.orchestra.run.smithi142.stdout:(6/6): python3-urwid-2.1.2-4.el9.x86_64.rpm 4.3 MB/s | 837 kB 00:00 2024-09-06T14:15:29.894 INFO:teuthology.orchestra.run.smithi142.stdout:-------------------------------------------------------------------------------- 2024-09-06T14:15:29.894 INFO:teuthology.orchestra.run.smithi142.stdout:Total 4.1 MB/s | 2.0 MB 00:00 2024-09-06T14:15:29.967 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction check 2024-09-06T14:15:29.981 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction check succeeded. 2024-09-06T14:15:29.981 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction test 2024-09-06T14:15:29.982 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-09-06T14:15:29.983 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-09-06T14:15:30.079 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction test succeeded. 2024-09-06T14:15:30.080 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction 2024-09-06T14:15:30.386 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-09-06T14:15:30.474 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-urwid-2.1.2-4.el9.x86_64 1/6 2024-09-06T14:15:30.486 INFO:teuthology.orchestra.run.smithi142.stdout: Preparing : 1/1 2024-09-06T14:15:30.507 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-configshell-1:1.1.30-1.el9.noarch 2/6 2024-09-06T14:15:30.536 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-kmod-0.9-32.el9.x86_64 3/6 2024-09-06T14:15:30.571 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-urwid-2.1.2-4.el9.x86_64 1/6 2024-09-06T14:15:30.605 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-configshell-1:1.1.30-1.el9.noarch 2/6 2024-09-06T14:15:30.642 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-kmod-0.9-32.el9.x86_64 3/6 2024-09-06T14:15:30.746 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libnvme-1.9-3.el9.x86_64 4/6 2024-09-06T14:15:30.776 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : nvme-cli-2.9.1-6.el9.x86_64 5/6 2024-09-06T14:15:30.866 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : libnvme-1.9-3.el9.x86_64 4/6 2024-09-06T14:15:30.898 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : nvme-cli-2.9.1-6.el9.x86_64 5/6 2024-09-06T14:15:31.217 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: nvme-cli-2.9.1-6.el9.x86_64 5/6 2024-09-06T14:15:31.246 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : nvmetcli-0.7-3.el9.noarch 6/6 2024-09-06T14:15:31.351 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: nvme-cli-2.9.1-6.el9.x86_64 5/6 2024-09-06T14:15:31.379 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : nvmetcli-0.7-3.el9.noarch 6/6 2024-09-06T14:15:32.224 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: nvmetcli-0.7-3.el9.noarch 6/6 2024-09-06T14:15:32.225 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/6 2024-09-06T14:15:32.225 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/6 2024-09-06T14:15:32.225 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : nvmetcli-0.7-3.el9.noarch 3/6 2024-09-06T14:15:32.225 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-configshell-1:1.1.30-1.el9.noarch 4/6 2024-09-06T14:15:32.225 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-kmod-0.9-32.el9.x86_64 5/6 2024-09-06T14:15:32.311 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: nvmetcli-0.7-3.el9.noarch 6/6 2024-09-06T14:15:32.311 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/6 2024-09-06T14:15:32.311 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/6 2024-09-06T14:15:32.311 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : nvmetcli-0.7-3.el9.noarch 3/6 2024-09-06T14:15:32.311 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-configshell-1:1.1.30-1.el9.noarch 4/6 2024-09-06T14:15:32.311 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-kmod-0.9-32.el9.x86_64 5/6 2024-09-06T14:15:32.587 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-urwid-2.1.2-4.el9.x86_64 6/6 2024-09-06T14:15:32.588 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-06T14:15:32.588 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-09-06T14:15:32.588 INFO:teuthology.orchestra.run.smithi162.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-06T14:15:32.588 INFO:teuthology.orchestra.run.smithi162.stdout: nvmetcli-0.7-3.el9.noarch python3-configshell-1:1.1.30-1.el9.noarch 2024-09-06T14:15:32.588 INFO:teuthology.orchestra.run.smithi162.stdout: python3-kmod-0.9-32.el9.x86_64 python3-urwid-2.1.2-4.el9.x86_64 2024-09-06T14:15:32.588 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-06T14:15:32.588 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-09-06T14:15:32.632 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-urwid-2.1.2-4.el9.x86_64 6/6 2024-09-06T14:15:32.632 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:15:32.632 INFO:teuthology.orchestra.run.smithi142.stdout:Installed: 2024-09-06T14:15:32.632 INFO:teuthology.orchestra.run.smithi142.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-06T14:15:32.632 INFO:teuthology.orchestra.run.smithi142.stdout: nvmetcli-0.7-3.el9.noarch python3-configshell-1:1.1.30-1.el9.noarch 2024-09-06T14:15:32.632 INFO:teuthology.orchestra.run.smithi142.stdout: python3-kmod-0.9-32.el9.x86_64 python3-urwid-2.1.2-4.el9.x86_64 2024-09-06T14:15:32.632 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:15:32.632 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2024-09-06T14:15:32.891 DEBUG:teuthology.parallel:result is None 2024-09-06T14:15:32.921 DEBUG:teuthology.parallel:result is None 2024-09-06T14:15:32.921 INFO:teuthology.run_tasks:Running task install... 2024-09-06T14:15:32.930 DEBUG:teuthology.task.install:project ceph 2024-09-06T14:15:32.931 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '8b38c33cecf4b0b7a9b3116eb9089e21e444ac73'}, 'extra_packages': ['cephadm']} 2024-09-06T14:15:32.931 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '8b38c33cecf4b0b7a9b3116eb9089e21e444ac73'} 2024-09-06T14:15:32.931 INFO:teuthology.task.install:Using flavor: default 2024-09-06T14:15:32.936 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-06T14:15:32.936 INFO:teuthology.task.install:extra packages: [] 2024-09-06T14:15:32.937 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '8b38c33cecf4b0b7a9b3116eb9089e21e444ac73', 'tag': None, 'wait_for_package': False} 2024-09-06T14:15:32.937 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 2024-09-06T14:15:32.939 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '8b38c33cecf4b0b7a9b3116eb9089e21e444ac73', 'tag': None, 'wait_for_package': False} 2024-09-06T14:15:32.940 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 2024-09-06T14:15:33.088 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-04-1837/8b38c33cecf4b0b7a9b3116eb9089e21e444ac73/centos/9/flavors/default/ 2024-09-06T14:15:33.088 INFO:teuthology.task.install.rpm:Package version is 19.3.0-4704.g8b38c33c 2024-09-06T14:15:33.102 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-04-1837/8b38c33cecf4b0b7a9b3116eb9089e21e444ac73/centos/9/flavors/default/ 2024-09-06T14:15:33.102 INFO:teuthology.task.install.rpm:Package version is 19.3.0-4704.g8b38c33c 2024-09-06T14:15:33.208 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-04-1837/8b38c33cecf4b0b7a9b3116eb9089e21e444ac73/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-04-1837/8b38c33cecf4b0b7a9b3116eb9089e21e444ac73/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-04-1837/8b38c33cecf4b0b7a9b3116eb9089e21e444ac73/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-06T14:15:33.208 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-09-06T14:15:33.208 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-06T14:15:33.222 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-04-1837/8b38c33cecf4b0b7a9b3116eb9089e21e444ac73/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-04-1837/8b38c33cecf4b0b7a9b3116eb9089e21e444ac73/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-04-1837/8b38c33cecf4b0b7a9b3116eb9089e21e444ac73/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-06T14:15:33.222 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-09-06T14:15:33.222 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-06T14:15:33.235 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-06T14:15:33.236 DEBUG:teuthology.orchestra.run.smithi142:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/8b38c33cecf4b0b7a9b3116eb9089e21e444ac73/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-06T14:15:33.250 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-06T14:15:33.250 DEBUG:teuthology.orchestra.run.smithi162:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/8b38c33cecf4b0b7a9b3116eb9089e21e444ac73/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-06T14:15:33.302 DEBUG:teuthology.orchestra.run.smithi142:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-06T14:15:33.317 DEBUG:teuthology.orchestra.run.smithi162:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-06T14:15:33.376 DEBUG:teuthology.orchestra.run.smithi142:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-06T14:15:33.401 DEBUG:teuthology.orchestra.run.smithi162:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-06T14:15:33.440 INFO:teuthology.orchestra.run.smithi142.stdout:check_obsoletes = 1 2024-09-06T14:15:33.442 DEBUG:teuthology.orchestra.run.smithi142:> sudo yum clean all 2024-09-06T14:15:33.467 INFO:teuthology.orchestra.run.smithi162.stdout:check_obsoletes = 1 2024-09-06T14:15:33.468 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum clean all 2024-09-06T14:15:33.669 INFO:teuthology.orchestra.run.smithi142.stdout:45 files removed 2024-09-06T14:15:33.694 INFO:teuthology.orchestra.run.smithi162.stdout:45 files removed 2024-09-06T14:15:33.695 DEBUG:teuthology.orchestra.run.smithi142:> sudo yum -y install ceph-radosgw 2024-09-06T14:15:33.721 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-radosgw 2024-09-06T14:15:34.242 INFO:teuthology.orchestra.run.smithi142.stdout:ceph packages for x86_64 311 kB/s | 84 kB 00:00 2024-09-06T14:15:34.266 INFO:teuthology.orchestra.run.smithi162.stdout:ceph packages for x86_64 309 kB/s | 84 kB 00:00 2024-09-06T14:15:34.543 INFO:teuthology.orchestra.run.smithi142.stdout:ceph noarch packages 94 kB/s | 24 kB 00:00 2024-09-06T14:15:34.559 INFO:teuthology.orchestra.run.smithi162.stdout:ceph noarch packages 97 kB/s | 24 kB 00:00 2024-09-06T14:15:34.819 INFO:teuthology.orchestra.run.smithi142.stdout:ceph source packages 8.1 kB/s | 1.9 kB 00:00 2024-09-06T14:15:34.835 INFO:teuthology.orchestra.run.smithi162.stdout:ceph source packages 8.0 kB/s | 1.9 kB 00:00 2024-09-06T14:15:35.393 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 8.2 MB 00:00 2024-09-06T14:15:37.527 INFO:teuthology.orchestra.run.smithi142.stdout:CentOS Stream 9 - BaseOS 3.1 MB/s | 8.2 MB 00:02 2024-09-06T14:15:38.194 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - AppStream 12 MB/s | 20 MB 00:01 2024-09-06T14:15:43.044 INFO:teuthology.orchestra.run.smithi142.stdout:CentOS Stream 9 - AppStream 4.6 MB/s | 20 MB 00:04 2024-09-06T14:15:44.553 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - CRB 2.6 MB/s | 6.5 MB 00:02 2024-09-06T14:15:45.962 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - Extras packages 89 kB/s | 19 kB 00:00 2024-09-06T14:15:47.591 INFO:teuthology.orchestra.run.smithi162.stdout:Extra Packages for Enterprise Linux 15 MB/s | 23 MB 00:01 2024-09-06T14:15:47.737 INFO:teuthology.orchestra.run.smithi142.stdout:CentOS Stream 9 - CRB 7.4 MB/s | 6.5 MB 00:00 2024-09-06T14:15:49.821 INFO:teuthology.orchestra.run.smithi142.stdout:CentOS Stream 9 - Extras packages 21 kB/s | 19 kB 00:00 2024-09-06T14:15:51.525 INFO:teuthology.orchestra.run.smithi142.stdout:Extra Packages for Enterprise Linux 14 MB/s | 23 MB 00:01 2024-09-06T14:15:53.030 INFO:teuthology.orchestra.run.smithi162.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2024-09-06T14:15:55.399 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-09-06T14:15:55.401 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-06T14:15:55.401 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repository Size 2024-09-06T14:15:55.401 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-06T14:15:55.401 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-09-06T14:15:55.402 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-radosgw x86_64 2:19.3.0-4704.g8b38c33c.el9 ceph 11 M 2024-09-06T14:15:55.402 INFO:teuthology.orchestra.run.smithi162.stdout:Upgrading: 2024-09-06T14:15:55.402 INFO:teuthology.orchestra.run.smithi162.stdout: librados2 x86_64 2:19.3.0-4704.g8b38c33c.el9 ceph 3.5 M 2024-09-06T14:15:55.402 INFO:teuthology.orchestra.run.smithi162.stdout: librbd1 x86_64 2:19.3.0-4704.g8b38c33c.el9 ceph 3.2 M 2024-09-06T14:15:55.402 INFO:teuthology.orchestra.run.smithi162.stdout:Installing dependencies: 2024-09-06T14:15:55.402 INFO:teuthology.orchestra.run.smithi162.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-06T14:15:55.402 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-base x86_64 2:19.3.0-4704.g8b38c33c.el9 ceph 5.5 M 2024-09-06T14:15:55.402 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-common x86_64 2:19.3.0-4704.g8b38c33c.el9 ceph 22 M 2024-09-06T14:15:55.402 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-selinux x86_64 2:19.3.0-4704.g8b38c33c.el9 ceph 25 k 2024-09-06T14:15:55.402 INFO:teuthology.orchestra.run.smithi162.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-09-06T14:15:55.402 INFO:teuthology.orchestra.run.smithi162.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-06T14:15:55.402 INFO:teuthology.orchestra.run.smithi162.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-06T14:15:55.403 INFO:teuthology.orchestra.run.smithi162.stdout: libcephfs2 x86_64 2:19.3.0-4704.g8b38c33c.el9 ceph 756 k 2024-09-06T14:15:55.403 INFO:teuthology.orchestra.run.smithi162.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-06T14:15:55.403 INFO:teuthology.orchestra.run.smithi162.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-06T14:15:55.403 INFO:teuthology.orchestra.run.smithi162.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-06T14:15:55.403 INFO:teuthology.orchestra.run.smithi162.stdout: libradosstriper1 x86_64 2:19.3.0-4704.g8b38c33c.el9 ceph 508 k 2024-09-06T14:15:55.403 INFO:teuthology.orchestra.run.smithi162.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-06T14:15:55.403 INFO:teuthology.orchestra.run.smithi162.stdout: librgw2 x86_64 2:19.3.0-4704.g8b38c33c.el9 ceph 5.5 M 2024-09-06T14:15:55.403 INFO:teuthology.orchestra.run.smithi162.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-06T14:15:55.403 INFO:teuthology.orchestra.run.smithi162.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-06T14:15:55.403 INFO:teuthology.orchestra.run.smithi162.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-06T14:15:55.403 INFO:teuthology.orchestra.run.smithi162.stdout: python3-ceph-argparse x86_64 2:19.3.0-4704.g8b38c33c.el9 ceph 45 k 2024-09-06T14:15:55.403 INFO:teuthology.orchestra.run.smithi162.stdout: python3-ceph-common x86_64 2:19.3.0-4704.g8b38c33c.el9 ceph 145 k 2024-09-06T14:15:55.403 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cephfs x86_64 2:19.3.0-4704.g8b38c33c.el9 ceph 161 k 2024-09-06T14:15:55.403 INFO:teuthology.orchestra.run.smithi162.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-06T14:15:55.404 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rados x86_64 2:19.3.0-4704.g8b38c33c.el9 ceph 320 k 2024-09-06T14:15:55.404 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rbd x86_64 2:19.3.0-4704.g8b38c33c.el9 ceph 305 k 2024-09-06T14:15:55.404 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rgw x86_64 2:19.3.0-4704.g8b38c33c.el9 ceph 99 k 2024-09-06T14:15:55.404 INFO:teuthology.orchestra.run.smithi162.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-06T14:15:55.404 INFO:teuthology.orchestra.run.smithi162.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-06T14:15:55.404 INFO:teuthology.orchestra.run.smithi162.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-06T14:15:55.404 INFO:teuthology.orchestra.run.smithi162.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-06T14:15:55.404 INFO:teuthology.orchestra.run.smithi162.stdout:Installing weak dependencies: 2024-09-06T14:15:55.404 INFO:teuthology.orchestra.run.smithi162.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-06T14:15:55.404 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-06T14:15:55.404 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-09-06T14:15:55.404 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-06T14:15:55.404 INFO:teuthology.orchestra.run.smithi162.stdout:Install 30 Packages 2024-09-06T14:15:55.405 INFO:teuthology.orchestra.run.smithi162.stdout:Upgrade 2 Packages 2024-09-06T14:15:55.405 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-06T14:15:55.405 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 62 M 2024-09-06T14:15:55.405 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-09-06T14:15:56.281 INFO:teuthology.orchestra.run.smithi162.stdout:(1/32): ceph-base-19.3.0-4704.g8b38c33c.el9.x86 11 MB/s | 5.5 MB 00:00 2024-09-06T14:15:56.340 INFO:teuthology.orchestra.run.smithi162.stdout:(2/32): ceph-selinux-19.3.0-4704.g8b38c33c.el9. 433 kB/s | 25 kB 00:00 2024-09-06T14:15:56.449 INFO:teuthology.orchestra.run.smithi162.stdout:(3/32): libcephfs2-19.3.0-4704.g8b38c33c.el9.x8 6.8 MB/s | 756 kB 00:00 2024-09-06T14:15:56.574 INFO:teuthology.orchestra.run.smithi162.stdout:(4/32): ceph-radosgw-19.3.0-4704.g8b38c33c.el9. 13 MB/s | 11 MB 00:00 2024-09-06T14:15:56.599 INFO:teuthology.orchestra.run.smithi162.stdout:(5/32): libradosstriper1-19.3.0-4704.g8b38c33c. 3.3 MB/s | 508 kB 00:00 2024-09-06T14:15:56.666 INFO:teuthology.orchestra.run.smithi162.stdout:(6/32): python3-ceph-argparse-19.3.0-4704.g8b38 678 kB/s | 45 kB 00:00 2024-09-06T14:15:56.766 INFO:teuthology.orchestra.run.smithi162.stdout:(7/32): python3-ceph-common-19.3.0-4704.g8b38c3 1.4 MB/s | 145 kB 00:00 2024-09-06T14:15:56.842 INFO:teuthology.orchestra.run.smithi162.stdout:(8/32): librgw2-19.3.0-4704.g8b38c33c.el9.x86_6 21 MB/s | 5.5 MB 00:00 2024-09-06T14:15:56.866 INFO:teuthology.orchestra.run.smithi162.stdout:(9/32): python3-cephfs-19.3.0-4704.g8b38c33c.el 1.6 MB/s | 161 kB 00:00 2024-09-06T14:15:56.892 INFO:teuthology.orchestra.run.smithi162.stdout:(10/32): python3-rados-19.3.0-4704.g8b38c33c.el 6.3 MB/s | 320 kB 00:00 2024-09-06T14:15:56.905 INFO:teuthology.orchestra.run.smithi142.stdout:lab-extras 39 kB/s | 1.7 kB 00:00 2024-09-06T14:15:56.917 INFO:teuthology.orchestra.run.smithi162.stdout:(11/32): python3-rbd-19.3.0-4704.g8b38c33c.el9. 5.9 MB/s | 305 kB 00:00 2024-09-06T14:15:56.943 INFO:teuthology.orchestra.run.smithi162.stdout:(12/32): python3-rgw-19.3.0-4704.g8b38c33c.el9. 1.9 MB/s | 99 kB 00:00 2024-09-06T14:15:57.143 INFO:teuthology.orchestra.run.smithi162.stdout:(13/32): ceph-common-19.3.0-4704.g8b38c33c.el9. 16 MB/s | 22 MB 00:01 2024-09-06T14:15:57.193 INFO:teuthology.orchestra.run.smithi162.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 388 kB/s | 107 kB 00:00 2024-09-06T14:15:57.269 INFO:teuthology.orchestra.run.smithi162.stdout:(15/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 492 kB/s | 160 kB 00:00 2024-09-06T14:15:57.294 INFO:teuthology.orchestra.run.smithi162.stdout:(16/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 301 kB/s | 45 kB 00:00 2024-09-06T14:15:57.378 INFO:teuthology.orchestra.run.smithi162.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 508 kB/s | 42 kB 00:00 2024-09-06T14:15:57.403 INFO:teuthology.orchestra.run.smithi162.stdout:(18/32): librdkafka-1.6.1-102.el9.x86_64.rpm 3.1 MB/s | 662 kB 00:00 2024-09-06T14:15:57.428 INFO:teuthology.orchestra.run.smithi162.stdout:(19/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.8 MB/s | 292 kB 00:00 2024-09-06T14:15:57.479 INFO:teuthology.orchestra.run.smithi162.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 476 kB/s | 36 kB 00:00 2024-09-06T14:15:57.504 INFO:teuthology.orchestra.run.smithi162.stdout:(21/32): qatlib-24.02.0-1.el9.x86_64.rpm 1.7 MB/s | 221 kB 00:00 2024-09-06T14:15:57.529 INFO:teuthology.orchestra.run.smithi162.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 471 kB/s | 47 kB 00:00 2024-09-06T14:15:57.604 INFO:teuthology.orchestra.run.smithi162.stdout:(23/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 2.4 MB/s | 309 kB 00:00 2024-09-06T14:15:57.630 INFO:teuthology.orchestra.run.smithi162.stdout:(24/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 304 kB/s | 26 kB 00:00 2024-09-06T14:15:57.655 INFO:teuthology.orchestra.run.smithi162.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 969 kB/s | 49 kB 00:00 2024-09-06T14:15:57.680 INFO:teuthology.orchestra.run.smithi162.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-06T14:15:57.722 INFO:teuthology.orchestra.run.smithi162.stdout:(27/32): re2-20211101-20.el9.x86_64.rpm 4.5 MB/s | 191 kB 00:00 2024-09-06T14:15:57.756 INFO:teuthology.orchestra.run.smithi162.stdout:(28/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 8.1 MB/s | 839 kB 00:00 2024-09-06T14:15:57.881 INFO:teuthology.orchestra.run.smithi162.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 10 MB/s | 1.6 MB 00:00 2024-09-06T14:15:57.940 INFO:teuthology.orchestra.run.smithi162.stdout:(30/32): librados2-19.3.0-4704.g8b38c33c.el9.x8 19 MB/s | 3.5 MB 00:00 2024-09-06T14:15:58.007 INFO:teuthology.orchestra.run.smithi162.stdout:(31/32): libarrow-9.0.0-11.el9.x86_64.rpm 8.8 MB/s | 4.4 MB 00:00 2024-09-06T14:15:58.182 INFO:teuthology.orchestra.run.smithi162.stdout:(32/32): librbd1-19.3.0-4704.g8b38c33c.el9.x86_ 11 MB/s | 3.2 MB 00:00 2024-09-06T14:15:58.188 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-09-06T14:15:58.188 INFO:teuthology.orchestra.run.smithi162.stdout:Total 22 MB/s | 62 MB 00:02 2024-09-06T14:15:58.296 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-09-06T14:15:58.333 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-09-06T14:15:58.334 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-09-06T14:15:58.812 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-09-06T14:15:58.813 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-09-06T14:15:59.224 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2024-09-06T14:15:59.226 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-09-06T14:15:59.226 INFO:teuthology.orchestra.run.smithi142.stdout: Package Arch Version Repository Size 2024-09-06T14:15:59.226 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-09-06T14:15:59.226 INFO:teuthology.orchestra.run.smithi142.stdout:Installing: 2024-09-06T14:15:59.226 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-radosgw x86_64 2:19.3.0-4704.g8b38c33c.el9 ceph 11 M 2024-09-06T14:15:59.226 INFO:teuthology.orchestra.run.smithi142.stdout:Upgrading: 2024-09-06T14:15:59.226 INFO:teuthology.orchestra.run.smithi142.stdout: librados2 x86_64 2:19.3.0-4704.g8b38c33c.el9 ceph 3.5 M 2024-09-06T14:15:59.226 INFO:teuthology.orchestra.run.smithi142.stdout: librbd1 x86_64 2:19.3.0-4704.g8b38c33c.el9 ceph 3.2 M 2024-09-06T14:15:59.226 INFO:teuthology.orchestra.run.smithi142.stdout:Installing dependencies: 2024-09-06T14:15:59.226 INFO:teuthology.orchestra.run.smithi142.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-06T14:15:59.227 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-base x86_64 2:19.3.0-4704.g8b38c33c.el9 ceph 5.5 M 2024-09-06T14:15:59.227 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-common x86_64 2:19.3.0-4704.g8b38c33c.el9 ceph 22 M 2024-09-06T14:15:59.227 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-selinux x86_64 2:19.3.0-4704.g8b38c33c.el9 ceph 25 k 2024-09-06T14:15:59.227 INFO:teuthology.orchestra.run.smithi142.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-09-06T14:15:59.227 INFO:teuthology.orchestra.run.smithi142.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-06T14:15:59.227 INFO:teuthology.orchestra.run.smithi142.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-06T14:15:59.227 INFO:teuthology.orchestra.run.smithi142.stdout: libcephfs2 x86_64 2:19.3.0-4704.g8b38c33c.el9 ceph 756 k 2024-09-06T14:15:59.227 INFO:teuthology.orchestra.run.smithi142.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-06T14:15:59.227 INFO:teuthology.orchestra.run.smithi142.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-06T14:15:59.227 INFO:teuthology.orchestra.run.smithi142.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-06T14:15:59.227 INFO:teuthology.orchestra.run.smithi142.stdout: libradosstriper1 x86_64 2:19.3.0-4704.g8b38c33c.el9 ceph 508 k 2024-09-06T14:15:59.227 INFO:teuthology.orchestra.run.smithi142.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-06T14:15:59.227 INFO:teuthology.orchestra.run.smithi142.stdout: librgw2 x86_64 2:19.3.0-4704.g8b38c33c.el9 ceph 5.5 M 2024-09-06T14:15:59.228 INFO:teuthology.orchestra.run.smithi142.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-06T14:15:59.228 INFO:teuthology.orchestra.run.smithi142.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-06T14:15:59.228 INFO:teuthology.orchestra.run.smithi142.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-06T14:15:59.228 INFO:teuthology.orchestra.run.smithi142.stdout: python3-ceph-argparse x86_64 2:19.3.0-4704.g8b38c33c.el9 ceph 45 k 2024-09-06T14:15:59.228 INFO:teuthology.orchestra.run.smithi142.stdout: python3-ceph-common x86_64 2:19.3.0-4704.g8b38c33c.el9 ceph 145 k 2024-09-06T14:15:59.228 INFO:teuthology.orchestra.run.smithi142.stdout: python3-cephfs x86_64 2:19.3.0-4704.g8b38c33c.el9 ceph 161 k 2024-09-06T14:15:59.228 INFO:teuthology.orchestra.run.smithi142.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-06T14:15:59.228 INFO:teuthology.orchestra.run.smithi142.stdout: python3-rados x86_64 2:19.3.0-4704.g8b38c33c.el9 ceph 320 k 2024-09-06T14:15:59.228 INFO:teuthology.orchestra.run.smithi142.stdout: python3-rbd x86_64 2:19.3.0-4704.g8b38c33c.el9 ceph 305 k 2024-09-06T14:15:59.228 INFO:teuthology.orchestra.run.smithi142.stdout: python3-rgw x86_64 2:19.3.0-4704.g8b38c33c.el9 ceph 99 k 2024-09-06T14:15:59.228 INFO:teuthology.orchestra.run.smithi142.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-06T14:15:59.228 INFO:teuthology.orchestra.run.smithi142.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-06T14:15:59.228 INFO:teuthology.orchestra.run.smithi142.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-06T14:15:59.228 INFO:teuthology.orchestra.run.smithi142.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-06T14:15:59.229 INFO:teuthology.orchestra.run.smithi142.stdout:Installing weak dependencies: 2024-09-06T14:15:59.229 INFO:teuthology.orchestra.run.smithi142.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-06T14:15:59.229 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:15:59.229 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction Summary 2024-09-06T14:15:59.229 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-09-06T14:15:59.229 INFO:teuthology.orchestra.run.smithi142.stdout:Install 30 Packages 2024-09-06T14:15:59.229 INFO:teuthology.orchestra.run.smithi142.stdout:Upgrade 2 Packages 2024-09-06T14:15:59.229 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:15:59.230 INFO:teuthology.orchestra.run.smithi142.stdout:Total download size: 62 M 2024-09-06T14:15:59.230 INFO:teuthology.orchestra.run.smithi142.stdout:Downloading Packages: 2024-09-06T14:16:00.254 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-09-06T14:16:00.332 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-09-06T14:16:00.362 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-09-06T14:16:00.395 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-09-06T14:16:00.419 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-06T14:16:00.442 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-06T14:16:00.637 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-06T14:16:00.690 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-06T14:16:01.083 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-09-06T14:16:01.148 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : librados2-2:19.3.0-4704.g8b38c33c.el9.x86_64 7/34 2024-09-06T14:16:01.225 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: librados2-2:19.3.0-4704.g8b38c33c.el9.x86_64 7/34 2024-09-06T14:16:01.323 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-rados-2:19.3.0-4704.g8b38c33c.el9.x86_64 8/34 2024-09-06T14:16:01.375 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libcephfs2-2:19.3.0-4704.g8b38c33c.el9.x86_64 9/34 2024-09-06T14:16:01.398 INFO:teuthology.orchestra.run.smithi142.stdout:(1/32): ceph-radosgw-19.3.0-4704.g8b38c33c.el9. 6.1 MB/s | 11 MB 00:01 2024-09-06T14:16:01.438 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: libcephfs2-2:19.3.0-4704.g8b38c33c.el9.x86_64 9/34 2024-09-06T14:16:01.439 INFO:teuthology.orchestra.run.smithi142.stdout:(2/32): ceph-selinux-19.3.0-4704.g8b38c33c.el9. 606 kB/s | 25 kB 00:00 2024-09-06T14:16:01.484 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-09-06T14:16:01.527 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-ceph-argparse-2:19.3.0-4704.g8b38c33c.el9. 11/34 2024-09-06T14:16:01.597 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cephfs-2:19.3.0-4704.g8b38c33c.el9.x86_64 12/34 2024-09-06T14:16:01.627 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libradosstriper1-2:19.3.0-4704.g8b38c33c.el9.x86_6 13/34 2024-09-06T14:16:01.692 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: libradosstriper1-2:19.3.0-4704.g8b38c33c.el9.x86_6 13/34 2024-09-06T14:16:01.699 INFO:teuthology.orchestra.run.smithi142.stdout:(3/32): ceph-common-19.3.0-4704.g8b38c33c.el9.x 10 MB/s | 22 MB 00:02 2024-09-06T14:16:01.724 INFO:teuthology.orchestra.run.smithi142.stdout:(4/32): libcephfs2-19.3.0-4704.g8b38c33c.el9.x8 2.6 MB/s | 756 kB 00:00 2024-09-06T14:16:01.726 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-09-06T14:16:01.758 INFO:teuthology.orchestra.run.smithi142.stdout:(5/32): libradosstriper1-19.3.0-4704.g8b38c33c. 8.5 MB/s | 508 kB 00:00 2024-09-06T14:16:01.799 INFO:teuthology.orchestra.run.smithi142.stdout:(6/32): python3-ceph-argparse-19.3.0-4704.g8b38 1.1 MB/s | 45 kB 00:00 2024-09-06T14:16:01.837 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/34 2024-09-06T14:16:01.850 INFO:teuthology.orchestra.run.smithi142.stdout:(7/32): python3-ceph-common-19.3.0-4704.g8b38c3 2.8 MB/s | 145 kB 00:00 2024-09-06T14:16:01.869 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-09-06T14:16:01.909 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-09-06T14:16:01.916 INFO:teuthology.orchestra.run.smithi142.stdout:(8/32): python3-cephfs-19.3.0-4704.g8b38c33c.el 2.4 MB/s | 161 kB 00:00 2024-09-06T14:16:01.943 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-09-06T14:16:01.972 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-09-06T14:16:01.975 INFO:teuthology.orchestra.run.smithi142.stdout:(9/32): python3-rados-19.3.0-4704.g8b38c33c.el9 5.4 MB/s | 320 kB 00:00 2024-09-06T14:16:02.005 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-09-06T14:16:02.076 INFO:teuthology.orchestra.run.smithi142.stdout:(10/32): librgw2-19.3.0-4704.g8b38c33c.el9.x86_ 16 MB/s | 5.5 MB 00:00 2024-09-06T14:16:02.101 INFO:teuthology.orchestra.run.smithi142.stdout:(11/32): python3-rbd-19.3.0-4704.g8b38c33c.el9. 2.4 MB/s | 305 kB 00:00 2024-09-06T14:16:02.134 INFO:teuthology.orchestra.run.smithi142.stdout:(12/32): python3-rgw-19.3.0-4704.g8b38c33c.el9. 1.7 MB/s | 99 kB 00:00 2024-09-06T14:16:02.218 INFO:teuthology.orchestra.run.smithi142.stdout:(13/32): ceph-base-19.3.0-4704.g8b38c33c.el9.x8 2.1 MB/s | 5.5 MB 00:02 2024-09-06T14:16:02.324 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-09-06T14:16:02.327 INFO:teuthology.orchestra.run.smithi142.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 418 kB/s | 45 kB 00:00 2024-09-06T14:16:02.352 INFO:teuthology.orchestra.run.smithi142.stdout:(15/32): boost-program-options-1.75.0-8.el9.x86 426 kB/s | 107 kB 00:00 2024-09-06T14:16:02.367 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : librbd1-2:19.3.0-4704.g8b38c33c.el9.x86_64 22/34 2024-09-06T14:16:02.377 INFO:teuthology.orchestra.run.smithi142.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 660 kB/s | 160 kB 00:00 2024-09-06T14:16:02.428 INFO:teuthology.orchestra.run.smithi142.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 845 kB/s | 42 kB 00:00 2024-09-06T14:16:02.458 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: librbd1-2:19.3.0-4704.g8b38c33c.el9.x86_64 22/34 2024-09-06T14:16:02.486 INFO:teuthology.orchestra.run.smithi142.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.1 MB/s | 292 kB 00:00 2024-09-06T14:16:02.497 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-rbd-2:19.3.0-4704.g8b38c33c.el9.x86_64 23/34 2024-09-06T14:16:02.533 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-09-06T14:16:02.536 INFO:teuthology.orchestra.run.smithi142.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 2.0 MB/s | 221 kB 00:00 2024-09-06T14:16:02.562 INFO:teuthology.orchestra.run.smithi142.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 478 kB/s | 36 kB 00:00 2024-09-06T14:16:02.595 INFO:teuthology.orchestra.run.smithi142.stdout:(21/32): librdkafka-1.6.1-102.el9.x86_64.rpm 2.4 MB/s | 662 kB 00:00 2024-09-06T14:16:02.629 INFO:teuthology.orchestra.run.smithi142.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 710 kB/s | 47 kB 00:00 2024-09-06T14:16:02.679 INFO:teuthology.orchestra.run.smithi142.stdout:(23/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 2.6 MB/s | 309 kB 00:00 2024-09-06T14:16:02.813 INFO:teuthology.orchestra.run.smithi142.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 375 kB/s | 49 kB 00:00 2024-09-06T14:16:02.838 INFO:teuthology.orchestra.run.smithi142.stdout:(25/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 130 kB/s | 26 kB 00:00 2024-09-06T14:16:02.863 INFO:teuthology.orchestra.run.smithi142.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-06T14:16:02.888 INFO:teuthology.orchestra.run.smithi142.stdout:(27/32): re2-20211101-20.el9.x86_64.rpm 7.5 MB/s | 191 kB 00:00 2024-09-06T14:16:02.947 INFO:teuthology.orchestra.run.smithi142.stdout:(28/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 7.5 MB/s | 839 kB 00:00 2024-09-06T14:16:03.006 INFO:teuthology.orchestra.run.smithi142.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 14 MB/s | 1.6 MB 00:00 2024-09-06T14:16:03.023 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-09-06T14:16:03.053 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : librgw2-2:19.3.0-4704.g8b38c33c.el9.x86_64 26/34 2024-09-06T14:16:03.081 INFO:teuthology.orchestra.run.smithi142.stdout:(30/32): libarrow-9.0.0-11.el9.x86_64.rpm 9.2 MB/s | 4.4 MB 00:00 2024-09-06T14:16:03.124 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: librgw2-2:19.3.0-4704.g8b38c33c.el9.x86_64 26/34 2024-09-06T14:16:03.197 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-rgw-2:19.3.0-4704.g8b38c33c.el9.x86_64 27/34 2024-09-06T14:16:03.240 INFO:teuthology.orchestra.run.smithi142.stdout:(31/32): librbd1-19.3.0-4704.g8b38c33c.el9.x86_ 14 MB/s | 3.2 MB 00:00 2024-09-06T14:16:03.404 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-ceph-common-2:19.3.0-4704.g8b38c33c.el9.x8 28/34 2024-09-06T14:16:03.798 INFO:teuthology.orchestra.run.smithi142.stdout:(32/32): librados2-19.3.0-4704.g8b38c33c.el9.x8 4.1 MB/s | 3.5 MB 00:00 2024-09-06T14:16:03.804 INFO:teuthology.orchestra.run.smithi142.stdout:-------------------------------------------------------------------------------- 2024-09-06T14:16:03.804 INFO:teuthology.orchestra.run.smithi142.stdout:Total 13 MB/s | 62 MB 00:04 2024-09-06T14:16:03.906 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction check 2024-09-06T14:16:03.943 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction check succeeded. 2024-09-06T14:16:03.943 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction test 2024-09-06T14:16:04.423 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction test succeeded. 2024-09-06T14:16:04.424 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction 2024-09-06T14:16:05.724 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-common-2:19.3.0-4704.g8b38c33c.el9.x86_64 29/34 2024-09-06T14:16:05.779 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-common-2:19.3.0-4704.g8b38c33c.el9.x86_64 29/34 2024-09-06T14:16:05.855 INFO:teuthology.orchestra.run.smithi142.stdout: Preparing : 1/1 2024-09-06T14:16:05.923 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-09-06T14:16:05.961 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-09-06T14:16:05.995 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-09-06T14:16:06.018 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-06T14:16:06.036 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-06T14:16:06.194 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-06T14:16:06.247 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-06T14:16:06.335 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-common-2:19.3.0-4704.g8b38c33c.el9.x86_64 29/34 2024-09-06T14:16:06.382 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-base-2:19.3.0-4704.g8b38c33c.el9.x86_64 30/34 2024-09-06T14:16:06.473 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-base-2:19.3.0-4704.g8b38c33c.el9.x86_64 30/34 2024-09-06T14:16:06.473 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-06T14:16:06.474 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-06T14:16:06.474 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-06T14:16:06.508 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-selinux-2:19.3.0-4704.g8b38c33c.el9.x86_64 31/34 2024-09-06T14:16:06.641 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-09-06T14:16:06.697 INFO:teuthology.orchestra.run.smithi142.stdout: Upgrading : librados2-2:19.3.0-4704.g8b38c33c.el9.x86_64 7/34 2024-09-06T14:16:06.793 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: librados2-2:19.3.0-4704.g8b38c33c.el9.x86_64 7/34 2024-09-06T14:16:06.880 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-rados-2:19.3.0-4704.g8b38c33c.el9.x86_64 8/34 2024-09-06T14:16:06.907 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : libcephfs2-2:19.3.0-4704.g8b38c33c.el9.x86_64 9/34 2024-09-06T14:16:06.969 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: libcephfs2-2:19.3.0-4704.g8b38c33c.el9.x86_64 9/34 2024-09-06T14:16:06.999 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-09-06T14:16:07.043 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-ceph-argparse-2:19.3.0-4704.g8b38c33c.el9. 11/34 2024-09-06T14:16:07.112 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-cephfs-2:19.3.0-4704.g8b38c33c.el9.x86_64 12/34 2024-09-06T14:16:07.134 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : libradosstriper1-2:19.3.0-4704.g8b38c33c.el9.x86_6 13/34 2024-09-06T14:16:07.198 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: libradosstriper1-2:19.3.0-4704.g8b38c33c.el9.x86_6 13/34 2024-09-06T14:16:07.232 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-09-06T14:16:07.343 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/34 2024-09-06T14:16:07.375 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-09-06T14:16:07.415 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-09-06T14:16:07.440 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-09-06T14:16:07.469 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-09-06T14:16:07.502 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-09-06T14:16:07.828 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-09-06T14:16:08.032 INFO:teuthology.orchestra.run.smithi142.stdout: Upgrading : librbd1-2:19.3.0-4704.g8b38c33c.el9.x86_64 22/34 2024-09-06T14:16:08.122 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: librbd1-2:19.3.0-4704.g8b38c33c.el9.x86_64 22/34 2024-09-06T14:16:08.153 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-rbd-2:19.3.0-4704.g8b38c33c.el9.x86_64 23/34 2024-09-06T14:16:08.189 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-09-06T14:16:08.693 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-09-06T14:16:08.743 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : librgw2-2:19.3.0-4704.g8b38c33c.el9.x86_64 26/34 2024-09-06T14:16:08.805 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: librgw2-2:19.3.0-4704.g8b38c33c.el9.x86_64 26/34 2024-09-06T14:16:08.876 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-rgw-2:19.3.0-4704.g8b38c33c.el9.x86_64 27/34 2024-09-06T14:16:08.903 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-ceph-common-2:19.3.0-4704.g8b38c33c.el9.x8 28/34 2024-09-06T14:16:11.193 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-common-2:19.3.0-4704.g8b38c33c.el9.x86_64 29/34 2024-09-06T14:16:11.233 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-common-2:19.3.0-4704.g8b38c33c.el9.x86_64 29/34 2024-09-06T14:16:11.794 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-common-2:19.3.0-4704.g8b38c33c.el9.x86_64 29/34 2024-09-06T14:16:11.828 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-base-2:19.3.0-4704.g8b38c33c.el9.x86_64 30/34 2024-09-06T14:16:11.904 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-base-2:19.3.0-4704.g8b38c33c.el9.x86_64 30/34 2024-09-06T14:16:11.904 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-06T14:16:11.904 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-06T14:16:11.905 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:16:11.937 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-selinux-2:19.3.0-4704.g8b38c33c.el9.x86_64 31/34 2024-09-06T14:16:16.409 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-selinux-2:19.3.0-4704.g8b38c33c.el9.x86_64 31/34 2024-09-06T14:16:16.409 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /sys 2024-09-06T14:16:16.409 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /proc 2024-09-06T14:16:16.409 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /mnt 2024-09-06T14:16:16.409 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /var/tmp 2024-09-06T14:16:16.410 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /home 2024-09-06T14:16:16.410 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /root 2024-09-06T14:16:16.410 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /tmp 2024-09-06T14:16:16.410 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-06T14:16:17.387 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-radosgw-2:19.3.0-4704.g8b38c33c.el9.x86_64 32/34 2024-09-06T14:16:17.415 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-4704.g8b38c33c.el9.x86_64 32/34 2024-09-06T14:16:17.415 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-06T14:16:17.415 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-06T14:16:17.415 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-06T14:16:17.415 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-06T14:16:17.415 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-06T14:16:17.418 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-06T14:16:17.505 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-06T14:16:17.506 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-06T14:16:18.713 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-06T14:16:18.713 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-base-2:19.3.0-4704.g8b38c33c.el9.x86_64 1/34 2024-09-06T14:16:18.713 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-common-2:19.3.0-4704.g8b38c33c.el9.x86_64 2/34 2024-09-06T14:16:18.713 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-radosgw-2:19.3.0-4704.g8b38c33c.el9.x86_64 3/34 2024-09-06T14:16:18.713 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-selinux-2:19.3.0-4704.g8b38c33c.el9.x86_64 4/34 2024-09-06T14:16:18.713 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libcephfs2-2:19.3.0-4704.g8b38c33c.el9.x86_64 5/34 2024-09-06T14:16:18.713 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libradosstriper1-2:19.3.0-4704.g8b38c33c.el9.x86_6 6/34 2024-09-06T14:16:18.713 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librgw2-2:19.3.0-4704.g8b38c33c.el9.x86_64 7/34 2024-09-06T14:16:18.713 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-ceph-argparse-2:19.3.0-4704.g8b38c33c.el9. 8/34 2024-09-06T14:16:18.714 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-ceph-common-2:19.3.0-4704.g8b38c33c.el9.x8 9/34 2024-09-06T14:16:18.714 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cephfs-2:19.3.0-4704.g8b38c33c.el9.x86_64 10/34 2024-09-06T14:16:18.714 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-rados-2:19.3.0-4704.g8b38c33c.el9.x86_64 11/34 2024-09-06T14:16:18.714 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-rbd-2:19.3.0-4704.g8b38c33c.el9.x86_64 12/34 2024-09-06T14:16:18.714 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-rgw-2:19.3.0-4704.g8b38c33c.el9.x86_64 13/34 2024-09-06T14:16:18.714 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-09-06T14:16:18.714 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-09-06T14:16:18.714 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-09-06T14:16:18.714 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-09-06T14:16:18.714 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-09-06T14:16:18.714 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-09-06T14:16:18.714 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-09-06T14:16:18.714 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-09-06T14:16:18.714 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-09-06T14:16:18.715 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-09-06T14:16:18.715 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-09-06T14:16:18.715 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-09-06T14:16:18.715 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-09-06T14:16:18.715 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-09-06T14:16:18.715 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-09-06T14:16:18.715 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : re2-1:20211101-20.el9.x86_64 29/34 2024-09-06T14:16:18.715 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-09-06T14:16:18.715 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librados2-2:19.3.0-4704.g8b38c33c.el9.x86_64 31/34 2024-09-06T14:16:18.715 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-09-06T14:16:18.715 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librbd1-2:19.3.0-4704.g8b38c33c.el9.x86_64 33/34 2024-09-06T14:16:19.728 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-09-06T14:16:19.728 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-06T14:16:19.728 INFO:teuthology.orchestra.run.smithi162.stdout:Upgraded: 2024-09-06T14:16:19.728 INFO:teuthology.orchestra.run.smithi162.stdout: librados2-2:19.3.0-4704.g8b38c33c.el9.x86_64 2024-09-06T14:16:19.728 INFO:teuthology.orchestra.run.smithi162.stdout: librbd1-2:19.3.0-4704.g8b38c33c.el9.x86_64 2024-09-06T14:16:19.728 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-09-06T14:16:19.728 INFO:teuthology.orchestra.run.smithi162.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-06T14:16:19.729 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-base-2:19.3.0-4704.g8b38c33c.el9.x86_64 2024-09-06T14:16:19.729 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-common-2:19.3.0-4704.g8b38c33c.el9.x86_64 2024-09-06T14:16:19.729 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-radosgw-2:19.3.0-4704.g8b38c33c.el9.x86_64 2024-09-06T14:16:19.729 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-selinux-2:19.3.0-4704.g8b38c33c.el9.x86_64 2024-09-06T14:16:19.729 INFO:teuthology.orchestra.run.smithi162.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-09-06T14:16:19.729 INFO:teuthology.orchestra.run.smithi162.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-06T14:16:19.729 INFO:teuthology.orchestra.run.smithi162.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-06T14:16:19.729 INFO:teuthology.orchestra.run.smithi162.stdout: libcephfs2-2:19.3.0-4704.g8b38c33c.el9.x86_64 2024-09-06T14:16:19.729 INFO:teuthology.orchestra.run.smithi162.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-06T14:16:19.729 INFO:teuthology.orchestra.run.smithi162.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-06T14:16:19.730 INFO:teuthology.orchestra.run.smithi162.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-06T14:16:19.730 INFO:teuthology.orchestra.run.smithi162.stdout: libradosstriper1-2:19.3.0-4704.g8b38c33c.el9.x86_64 2024-09-06T14:16:19.730 INFO:teuthology.orchestra.run.smithi162.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-06T14:16:19.730 INFO:teuthology.orchestra.run.smithi162.stdout: librgw2-2:19.3.0-4704.g8b38c33c.el9.x86_64 2024-09-06T14:16:19.730 INFO:teuthology.orchestra.run.smithi162.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-06T14:16:19.730 INFO:teuthology.orchestra.run.smithi162.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-06T14:16:19.730 INFO:teuthology.orchestra.run.smithi162.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-06T14:16:19.730 INFO:teuthology.orchestra.run.smithi162.stdout: python3-ceph-argparse-2:19.3.0-4704.g8b38c33c.el9.x86_64 2024-09-06T14:16:19.730 INFO:teuthology.orchestra.run.smithi162.stdout: python3-ceph-common-2:19.3.0-4704.g8b38c33c.el9.x86_64 2024-09-06T14:16:19.730 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cephfs-2:19.3.0-4704.g8b38c33c.el9.x86_64 2024-09-06T14:16:19.730 INFO:teuthology.orchestra.run.smithi162.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-06T14:16:19.730 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rados-2:19.3.0-4704.g8b38c33c.el9.x86_64 2024-09-06T14:16:19.731 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rbd-2:19.3.0-4704.g8b38c33c.el9.x86_64 2024-09-06T14:16:19.731 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rgw-2:19.3.0-4704.g8b38c33c.el9.x86_64 2024-09-06T14:16:19.731 INFO:teuthology.orchestra.run.smithi162.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-06T14:16:19.731 INFO:teuthology.orchestra.run.smithi162.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-06T14:16:19.731 INFO:teuthology.orchestra.run.smithi162.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-06T14:16:19.731 INFO:teuthology.orchestra.run.smithi162.stdout: re2-1:20211101-20.el9.x86_64 2024-09-06T14:16:19.731 INFO:teuthology.orchestra.run.smithi162.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-06T14:16:19.731 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-06T14:16:19.731 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-09-06T14:16:20.140 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-test 2024-09-06T14:16:20.697 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:00:28 ago on Fri 06 Sep 2024 02:15:52 PM UTC. 2024-09-06T14:16:20.829 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-09-06T14:16:20.829 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-06T14:16:20.830 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repository Size 2024-09-06T14:16:20.830 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-06T14:16:20.830 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-09-06T14:16:20.830 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-test x86_64 2:19.3.0-4704.g8b38c33c.el9 ceph 50 M 2024-09-06T14:16:20.830 INFO:teuthology.orchestra.run.smithi162.stdout:Installing dependencies: 2024-09-06T14:16:20.830 INFO:teuthology.orchestra.run.smithi162.stdout: libcephsqlite x86_64 2:19.3.0-4704.g8b38c33c.el9 ceph 161 k 2024-09-06T14:16:20.830 INFO:teuthology.orchestra.run.smithi162.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-06T14:16:20.830 INFO:teuthology.orchestra.run.smithi162.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-06T14:16:20.830 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-06T14:16:20.830 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-09-06T14:16:20.830 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-06T14:16:20.830 INFO:teuthology.orchestra.run.smithi162.stdout:Install 4 Packages 2024-09-06T14:16:20.831 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-06T14:16:20.831 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 51 M 2024-09-06T14:16:20.831 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 216 M 2024-09-06T14:16:20.831 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-09-06T14:16:21.151 INFO:teuthology.orchestra.run.smithi162.stdout:(1/4): libcephsqlite-19.3.0-4704.g8b38c33c.el9. 735 kB/s | 161 kB 00:00 2024-09-06T14:16:21.176 INFO:teuthology.orchestra.run.smithi162.stdout:(2/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.2 MB/s | 304 kB 00:00 2024-09-06T14:16:21.302 INFO:teuthology.orchestra.run.smithi162.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 424 kB/s | 64 kB 00:00 2024-09-06T14:16:21.948 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-selinux-2:19.3.0-4704.g8b38c33c.el9.x86_64 31/34 2024-09-06T14:16:21.948 INFO:teuthology.orchestra.run.smithi142.stdout:skipping the directory /sys 2024-09-06T14:16:21.948 INFO:teuthology.orchestra.run.smithi142.stdout:skipping the directory /proc 2024-09-06T14:16:21.948 INFO:teuthology.orchestra.run.smithi142.stdout:skipping the directory /mnt 2024-09-06T14:16:21.948 INFO:teuthology.orchestra.run.smithi142.stdout:skipping the directory /var/tmp 2024-09-06T14:16:21.948 INFO:teuthology.orchestra.run.smithi142.stdout:skipping the directory /home 2024-09-06T14:16:21.949 INFO:teuthology.orchestra.run.smithi142.stdout:skipping the directory /root 2024-09-06T14:16:21.949 INFO:teuthology.orchestra.run.smithi142.stdout:skipping the directory /tmp 2024-09-06T14:16:21.949 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:16:22.552 INFO:teuthology.orchestra.run.smithi162.stdout:(4/4): ceph-test-19.3.0-4704.g8b38c33c.el9.x86_ 31 MB/s | 50 MB 00:01 2024-09-06T14:16:22.554 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-09-06T14:16:22.554 INFO:teuthology.orchestra.run.smithi162.stdout:Total 29 MB/s | 51 MB 00:01 2024-09-06T14:16:22.583 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-09-06T14:16:22.603 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-09-06T14:16:22.603 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-09-06T14:16:22.981 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-09-06T14:16:22.981 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-09-06T14:16:22.991 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-radosgw-2:19.3.0-4704.g8b38c33c.el9.x86_64 32/34 2024-09-06T14:16:23.016 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-4704.g8b38c33c.el9.x86_64 32/34 2024-09-06T14:16:23.016 INFO:teuthology.orchestra.run.smithi142.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-06T14:16:23.016 INFO:teuthology.orchestra.run.smithi142.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-06T14:16:23.016 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-06T14:16:23.016 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-06T14:16:23.016 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:16:23.018 INFO:teuthology.orchestra.run.smithi142.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-06T14:16:23.101 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-06T14:16:23.101 INFO:teuthology.orchestra.run.smithi142.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-06T14:16:23.520 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-09-06T14:16:23.603 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-06T14:16:23.642 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-06T14:16:23.671 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libcephsqlite-2:19.3.0-4704.g8b38c33c.el9.x86_64 3/4 2024-09-06T14:16:24.258 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-06T14:16:24.258 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-base-2:19.3.0-4704.g8b38c33c.el9.x86_64 1/34 2024-09-06T14:16:24.258 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-common-2:19.3.0-4704.g8b38c33c.el9.x86_64 2/34 2024-09-06T14:16:24.258 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-radosgw-2:19.3.0-4704.g8b38c33c.el9.x86_64 3/34 2024-09-06T14:16:24.258 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-selinux-2:19.3.0-4704.g8b38c33c.el9.x86_64 4/34 2024-09-06T14:16:24.258 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libcephfs2-2:19.3.0-4704.g8b38c33c.el9.x86_64 5/34 2024-09-06T14:16:24.258 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libradosstriper1-2:19.3.0-4704.g8b38c33c.el9.x86_6 6/34 2024-09-06T14:16:24.259 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : librgw2-2:19.3.0-4704.g8b38c33c.el9.x86_64 7/34 2024-09-06T14:16:24.259 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-ceph-argparse-2:19.3.0-4704.g8b38c33c.el9. 8/34 2024-09-06T14:16:24.259 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-ceph-common-2:19.3.0-4704.g8b38c33c.el9.x8 9/34 2024-09-06T14:16:24.259 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-cephfs-2:19.3.0-4704.g8b38c33c.el9.x86_64 10/34 2024-09-06T14:16:24.259 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-rados-2:19.3.0-4704.g8b38c33c.el9.x86_64 11/34 2024-09-06T14:16:24.259 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-rbd-2:19.3.0-4704.g8b38c33c.el9.x86_64 12/34 2024-09-06T14:16:24.259 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-rgw-2:19.3.0-4704.g8b38c33c.el9.x86_64 13/34 2024-09-06T14:16:24.259 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-09-06T14:16:24.259 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-09-06T14:16:24.259 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-09-06T14:16:24.259 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-09-06T14:16:24.259 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-09-06T14:16:24.259 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-09-06T14:16:24.260 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-09-06T14:16:24.260 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-09-06T14:16:24.260 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-09-06T14:16:24.260 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-09-06T14:16:24.260 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-09-06T14:16:24.260 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-09-06T14:16:24.260 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-09-06T14:16:24.260 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-09-06T14:16:24.260 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-09-06T14:16:24.260 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : re2-1:20211101-20.el9.x86_64 29/34 2024-09-06T14:16:24.260 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-09-06T14:16:24.260 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : librados2-2:19.3.0-4704.g8b38c33c.el9.x86_64 31/34 2024-09-06T14:16:24.260 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-09-06T14:16:24.261 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : librbd1-2:19.3.0-4704.g8b38c33c.el9.x86_64 33/34 2024-09-06T14:16:25.170 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-09-06T14:16:25.171 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:16:25.171 INFO:teuthology.orchestra.run.smithi142.stdout:Upgraded: 2024-09-06T14:16:25.171 INFO:teuthology.orchestra.run.smithi142.stdout: librados2-2:19.3.0-4704.g8b38c33c.el9.x86_64 2024-09-06T14:16:25.171 INFO:teuthology.orchestra.run.smithi142.stdout: librbd1-2:19.3.0-4704.g8b38c33c.el9.x86_64 2024-09-06T14:16:25.171 INFO:teuthology.orchestra.run.smithi142.stdout:Installed: 2024-09-06T14:16:25.171 INFO:teuthology.orchestra.run.smithi142.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-06T14:16:25.171 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-base-2:19.3.0-4704.g8b38c33c.el9.x86_64 2024-09-06T14:16:25.171 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-common-2:19.3.0-4704.g8b38c33c.el9.x86_64 2024-09-06T14:16:25.171 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-radosgw-2:19.3.0-4704.g8b38c33c.el9.x86_64 2024-09-06T14:16:25.171 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-selinux-2:19.3.0-4704.g8b38c33c.el9.x86_64 2024-09-06T14:16:25.171 INFO:teuthology.orchestra.run.smithi142.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-09-06T14:16:25.171 INFO:teuthology.orchestra.run.smithi142.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-06T14:16:25.172 INFO:teuthology.orchestra.run.smithi142.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-06T14:16:25.172 INFO:teuthology.orchestra.run.smithi142.stdout: libcephfs2-2:19.3.0-4704.g8b38c33c.el9.x86_64 2024-09-06T14:16:25.172 INFO:teuthology.orchestra.run.smithi142.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-06T14:16:25.172 INFO:teuthology.orchestra.run.smithi142.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-06T14:16:25.172 INFO:teuthology.orchestra.run.smithi142.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-06T14:16:25.172 INFO:teuthology.orchestra.run.smithi142.stdout: libradosstriper1-2:19.3.0-4704.g8b38c33c.el9.x86_64 2024-09-06T14:16:25.172 INFO:teuthology.orchestra.run.smithi142.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-06T14:16:25.172 INFO:teuthology.orchestra.run.smithi142.stdout: librgw2-2:19.3.0-4704.g8b38c33c.el9.x86_64 2024-09-06T14:16:25.172 INFO:teuthology.orchestra.run.smithi142.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-06T14:16:25.172 INFO:teuthology.orchestra.run.smithi142.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-06T14:16:25.172 INFO:teuthology.orchestra.run.smithi142.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-06T14:16:25.172 INFO:teuthology.orchestra.run.smithi142.stdout: python3-ceph-argparse-2:19.3.0-4704.g8b38c33c.el9.x86_64 2024-09-06T14:16:25.172 INFO:teuthology.orchestra.run.smithi142.stdout: python3-ceph-common-2:19.3.0-4704.g8b38c33c.el9.x86_64 2024-09-06T14:16:25.172 INFO:teuthology.orchestra.run.smithi142.stdout: python3-cephfs-2:19.3.0-4704.g8b38c33c.el9.x86_64 2024-09-06T14:16:25.173 INFO:teuthology.orchestra.run.smithi142.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-06T14:16:25.173 INFO:teuthology.orchestra.run.smithi142.stdout: python3-rados-2:19.3.0-4704.g8b38c33c.el9.x86_64 2024-09-06T14:16:25.173 INFO:teuthology.orchestra.run.smithi142.stdout: python3-rbd-2:19.3.0-4704.g8b38c33c.el9.x86_64 2024-09-06T14:16:25.173 INFO:teuthology.orchestra.run.smithi142.stdout: python3-rgw-2:19.3.0-4704.g8b38c33c.el9.x86_64 2024-09-06T14:16:25.173 INFO:teuthology.orchestra.run.smithi142.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-06T14:16:25.173 INFO:teuthology.orchestra.run.smithi142.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-06T14:16:25.173 INFO:teuthology.orchestra.run.smithi142.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-06T14:16:25.173 INFO:teuthology.orchestra.run.smithi142.stdout: re2-1:20211101-20.el9.x86_64 2024-09-06T14:16:25.173 INFO:teuthology.orchestra.run.smithi142.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-06T14:16:25.173 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:16:25.173 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2024-09-06T14:16:25.537 DEBUG:teuthology.orchestra.run.smithi142:> sudo yum -y install ceph-test 2024-09-06T14:16:26.097 INFO:teuthology.orchestra.run.smithi142.stdout:Last metadata expiration check: 0:00:30 ago on Fri 06 Sep 2024 02:15:56 PM UTC. 2024-09-06T14:16:26.233 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2024-09-06T14:16:26.234 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-09-06T14:16:26.234 INFO:teuthology.orchestra.run.smithi142.stdout: Package Arch Version Repository Size 2024-09-06T14:16:26.234 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-09-06T14:16:26.234 INFO:teuthology.orchestra.run.smithi142.stdout:Installing: 2024-09-06T14:16:26.234 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-test x86_64 2:19.3.0-4704.g8b38c33c.el9 ceph 50 M 2024-09-06T14:16:26.234 INFO:teuthology.orchestra.run.smithi142.stdout:Installing dependencies: 2024-09-06T14:16:26.234 INFO:teuthology.orchestra.run.smithi142.stdout: libcephsqlite x86_64 2:19.3.0-4704.g8b38c33c.el9 ceph 161 k 2024-09-06T14:16:26.234 INFO:teuthology.orchestra.run.smithi142.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-06T14:16:26.234 INFO:teuthology.orchestra.run.smithi142.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-06T14:16:26.234 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:16:26.235 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction Summary 2024-09-06T14:16:26.235 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-09-06T14:16:26.235 INFO:teuthology.orchestra.run.smithi142.stdout:Install 4 Packages 2024-09-06T14:16:26.235 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:16:26.235 INFO:teuthology.orchestra.run.smithi142.stdout:Total download size: 51 M 2024-09-06T14:16:26.235 INFO:teuthology.orchestra.run.smithi142.stdout:Installed size: 216 M 2024-09-06T14:16:26.236 INFO:teuthology.orchestra.run.smithi142.stdout:Downloading Packages: 2024-09-06T14:16:26.496 INFO:teuthology.orchestra.run.smithi142.stdout:(1/4): libcephsqlite-19.3.0-4704.g8b38c33c.el9. 685 kB/s | 161 kB 00:00 2024-09-06T14:16:26.847 INFO:teuthology.orchestra.run.smithi142.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 182 kB/s | 64 kB 00:00 2024-09-06T14:16:26.872 INFO:teuthology.orchestra.run.smithi142.stdout:(3/4): socat-1.7.4.1-6.el9.x86_64.rpm 498 kB/s | 304 kB 00:00 2024-09-06T14:16:27.839 INFO:teuthology.orchestra.run.smithi142.stdout:(4/4): ceph-test-19.3.0-4704.g8b38c33c.el9.x86_ 32 MB/s | 50 MB 00:01 2024-09-06T14:16:27.841 INFO:teuthology.orchestra.run.smithi142.stdout:-------------------------------------------------------------------------------- 2024-09-06T14:16:27.842 INFO:teuthology.orchestra.run.smithi142.stdout:Total 32 MB/s | 51 MB 00:01 2024-09-06T14:16:27.869 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction check 2024-09-06T14:16:27.889 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction check succeeded. 2024-09-06T14:16:27.889 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction test 2024-09-06T14:16:28.267 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction test succeeded. 2024-09-06T14:16:28.268 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction 2024-09-06T14:16:28.358 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: libcephsqlite-2:19.3.0-4704.g8b38c33c.el9.x86_64 3/4 2024-09-06T14:16:28.451 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-test-2:19.3.0-4704.g8b38c33c.el9.x86_64 4/4 2024-09-06T14:16:28.800 INFO:teuthology.orchestra.run.smithi142.stdout: Preparing : 1/1 2024-09-06T14:16:28.882 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-06T14:16:28.929 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-06T14:16:28.966 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : libcephsqlite-2:19.3.0-4704.g8b38c33c.el9.x86_64 3/4 2024-09-06T14:16:30.925 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-test-2:19.3.0-4704.g8b38c33c.el9.x86_64 4/4 2024-09-06T14:16:30.926 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-test-2:19.3.0-4704.g8b38c33c.el9.x86_64 1/4 2024-09-06T14:16:30.926 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libcephsqlite-2:19.3.0-4704.g8b38c33c.el9.x86_64 2/4 2024-09-06T14:16:30.926 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-06T14:16:31.225 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-06T14:16:31.225 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-06T14:16:31.225 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-09-06T14:16:31.225 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-test-2:19.3.0-4704.g8b38c33c.el9.x86_64 2024-09-06T14:16:31.225 INFO:teuthology.orchestra.run.smithi162.stdout: libcephsqlite-2:19.3.0-4704.g8b38c33c.el9.x86_64 2024-09-06T14:16:31.226 INFO:teuthology.orchestra.run.smithi162.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-06T14:16:31.226 INFO:teuthology.orchestra.run.smithi162.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-06T14:16:31.226 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-06T14:16:31.226 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-09-06T14:16:31.538 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph 2024-09-06T14:16:32.113 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:00:40 ago on Fri 06 Sep 2024 02:15:52 PM UTC. 2024-09-06T14:16:32.267 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-09-06T14:16:32.270 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-06T14:16:32.270 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repository Size 2024-09-06T14:16:32.270 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-06T14:16:32.270 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-09-06T14:16:32.270 INFO:teuthology.orchestra.run.smithi162.stdout: ceph x86_64 2:19.3.0-4704.g8b38c33c.el9 ceph 6.6 k 2024-09-06T14:16:32.270 INFO:teuthology.orchestra.run.smithi162.stdout:Installing dependencies: 2024-09-06T14:16:32.271 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mds x86_64 2:19.3.0-4704.g8b38c33c.el9 ceph 2.4 M 2024-09-06T14:16:32.271 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr x86_64 2:19.3.0-4704.g8b38c33c.el9 ceph 1.5 M 2024-09-06T14:16:32.271 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-modules-core noarch 2:19.3.0-4704.g8b38c33c.el9 ceph-noarch 291 k 2024-09-06T14:16:32.271 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mon x86_64 2:19.3.0-4704.g8b38c33c.el9 ceph 4.8 M 2024-09-06T14:16:32.271 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-osd x86_64 2:19.3.0-4704.g8b38c33c.el9 ceph 17 M 2024-09-06T14:16:32.271 INFO:teuthology.orchestra.run.smithi162.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-06T14:16:32.271 INFO:teuthology.orchestra.run.smithi162.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-06T14:16:32.271 INFO:teuthology.orchestra.run.smithi162.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-06T14:16:32.271 INFO:teuthology.orchestra.run.smithi162.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-06T14:16:32.271 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-06T14:16:32.271 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-06T14:16:32.271 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-06T14:16:32.271 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-06T14:16:32.271 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-06T14:16:32.272 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-06T14:16:32.272 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-collections 2024-09-06T14:16:32.272 INFO:teuthology.orchestra.run.smithi162.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-06T14:16:32.272 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-06T14:16:32.272 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-06T14:16:32.272 INFO:teuthology.orchestra.run.smithi162.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-06T14:16:32.272 INFO:teuthology.orchestra.run.smithi162.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-06T14:16:32.272 INFO:teuthology.orchestra.run.smithi162.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-06T14:16:32.272 INFO:teuthology.orchestra.run.smithi162.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-06T14:16:32.272 INFO:teuthology.orchestra.run.smithi162.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-06T14:16:32.272 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-06T14:16:32.273 INFO:teuthology.orchestra.run.smithi162.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-06T14:16:32.273 INFO:teuthology.orchestra.run.smithi162.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-06T14:16:32.273 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-06T14:16:32.273 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-06T14:16:32.273 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-06T14:16:32.273 INFO:teuthology.orchestra.run.smithi162.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-06T14:16:32.273 INFO:teuthology.orchestra.run.smithi162.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-06T14:16:32.273 INFO:teuthology.orchestra.run.smithi162.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-06T14:16:32.273 INFO:teuthology.orchestra.run.smithi162.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-06T14:16:32.273 INFO:teuthology.orchestra.run.smithi162.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-06T14:16:32.273 INFO:teuthology.orchestra.run.smithi162.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-06T14:16:32.273 INFO:teuthology.orchestra.run.smithi162.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-06T14:16:32.273 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-06T14:16:32.274 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-09-06T14:16:32.274 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-06T14:16:32.274 INFO:teuthology.orchestra.run.smithi162.stdout:Install 37 Packages 2024-09-06T14:16:32.274 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-06T14:16:32.274 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 31 M 2024-09-06T14:16:32.274 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 109 M 2024-09-06T14:16:32.275 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-09-06T14:16:32.880 INFO:teuthology.orchestra.run.smithi162.stdout:(1/37): ceph-19.3.0-4704.g8b38c33c.el9.x86_64.r 52 kB/s | 6.6 kB 00:00 2024-09-06T14:16:33.063 INFO:teuthology.orchestra.run.smithi162.stdout:(2/37): ceph-mgr-19.3.0-4704.g8b38c33c.el9.x86_ 4.9 MB/s | 1.5 MB 00:00 2024-09-06T14:16:33.113 INFO:teuthology.orchestra.run.smithi162.stdout:(3/37): ceph-mds-19.3.0-4704.g8b38c33c.el9.x86_ 6.6 MB/s | 2.4 MB 00:00 2024-09-06T14:16:33.206 INFO:teuthology.orchestra.run.smithi162.stdout:(4/37): ceph-mon-19.3.0-4704.g8b38c33c.el9.x86_ 15 MB/s | 4.8 MB 00:00 2024-09-06T14:16:33.231 INFO:teuthology.orchestra.run.smithi162.stdout:(5/37): ceph-mgr-modules-core-19.3.0-4704.g8b38 2.4 MB/s | 291 kB 00:00 2024-09-06T14:16:33.440 INFO:teuthology.orchestra.run.smithi162.stdout:(6/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.1 MB/s | 253 kB 00:00 2024-09-06T14:16:33.632 INFO:teuthology.orchestra.run.smithi162.stdout:(7/37): ceph-osd-19.3.0-4704.g8b38c33c.el9.x86_ 30 MB/s | 17 MB 00:00 2024-09-06T14:16:33.657 INFO:teuthology.orchestra.run.smithi162.stdout:(8/37): python3-ply-3.11-14.el9.noarch.rpm 491 kB/s | 106 kB 00:00 2024-09-06T14:16:33.664 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: libcephsqlite-2:19.3.0-4704.g8b38c33c.el9.x86_64 3/4 2024-09-06T14:16:33.707 INFO:teuthology.orchestra.run.smithi162.stdout:(9/37): python3-requests-2.25.1-8.el9.noarch.rp 2.4 MB/s | 125 kB 00:00 2024-09-06T14:16:33.741 INFO:teuthology.orchestra.run.smithi162.stdout:(10/37): python3-cryptography-36.0.1-4.el9.x86_ 2.4 MB/s | 1.2 MB 00:00 2024-09-06T14:16:33.747 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-test-2:19.3.0-4704.g8b38c33c.el9.x86_64 4/4 2024-09-06T14:16:33.767 INFO:teuthology.orchestra.run.smithi162.stdout:(11/37): python3-urllib3-1.26.5-6.el9.noarch.rp 3.6 MB/s | 215 kB 00:00 2024-09-06T14:16:33.800 INFO:teuthology.orchestra.run.smithi162.stdout:(12/37): python3-pycparser-2.20-6.el9.noarch.rp 805 kB/s | 135 kB 00:00 2024-09-06T14:16:33.909 INFO:teuthology.orchestra.run.smithi162.stdout:(13/37): lua-5.4.4-4.el9.x86_64.rpm 1.1 MB/s | 188 kB 00:00 2024-09-06T14:16:33.934 INFO:teuthology.orchestra.run.smithi162.stdout:(14/37): python3-markupsafe-1.1.1-12.el9.x86_64 260 kB/s | 35 kB 00:00 2024-09-06T14:16:33.959 INFO:teuthology.orchestra.run.smithi162.stdout:(15/37): python3-mako-1.1.4-6.el9.noarch.rpm 895 kB/s | 172 kB 00:00 2024-09-06T14:16:33.984 INFO:teuthology.orchestra.run.smithi162.stdout:(16/37): python3-packaging-20.9-5.el9.noarch.rp 1.0 MB/s | 77 kB 00:00 2024-09-06T14:16:34.010 INFO:teuthology.orchestra.run.smithi162.stdout:(17/37): python3-toml-0.10.2-6.el9.noarch.rpm 825 kB/s | 42 kB 00:00 2024-09-06T14:16:34.035 INFO:teuthology.orchestra.run.smithi162.stdout:(18/37): python3-pytz-2021.1-5.el9.noarch.rpm 504 kB/s | 51 kB 00:00 2024-09-06T14:16:34.094 INFO:teuthology.orchestra.run.smithi162.stdout:(19/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 745 kB/s | 43 kB 00:00 2024-09-06T14:16:34.119 INFO:teuthology.orchestra.run.smithi162.stdout:(20/37): luarocks-3.9.2-5.el9.noarch.rpm 1.4 MB/s | 151 kB 00:00 2024-09-06T14:16:34.153 INFO:teuthology.orchestra.run.smithi162.stdout:(21/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.9 MB/s | 172 kB 00:00 2024-09-06T14:16:34.178 INFO:teuthology.orchestra.run.smithi162.stdout:(22/37): lua-devel-5.4.4-4.el9.x86_64.rpm 115 kB/s | 22 kB 00:00 2024-09-06T14:16:34.203 INFO:teuthology.orchestra.run.smithi162.stdout:(23/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 212 kB/s | 11 kB 00:00 2024-09-06T14:16:34.237 INFO:teuthology.orchestra.run.smithi162.stdout:(24/37): python3-cherrypy-18.6.1-2.el9.noarch.r 3.0 MB/s | 358 kB 00:00 2024-09-06T14:16:34.263 INFO:teuthology.orchestra.run.smithi162.stdout:(25/37): python3-jaraco-collections-3.0.0-8.el9 449 kB/s | 23 kB 00:00 2024-09-06T14:16:34.287 INFO:teuthology.orchestra.run.smithi162.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 386 kB/s | 19 kB 00:00 2024-09-06T14:16:34.313 INFO:teuthology.orchestra.run.smithi162.stdout:(27/37): python3-jaraco-classes-3.2.1-5.el9.noa 131 kB/s | 18 kB 00:00 2024-09-06T14:16:34.338 INFO:teuthology.orchestra.run.smithi162.stdout:(28/37): python3-jaraco-text-3.2.0-6.el9.noarch 267 kB/s | 20 kB 00:00 2024-09-06T14:16:34.363 INFO:teuthology.orchestra.run.smithi162.stdout:(29/37): python3-logutils-0.3.5-21.el9.noarch.r 613 kB/s | 46 kB 00:00 2024-09-06T14:16:34.389 INFO:teuthology.orchestra.run.smithi162.stdout:(30/37): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-09-06T14:16:34.414 INFO:teuthology.orchestra.run.smithi162.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 328 kB/s | 16 kB 00:00 2024-09-06T14:16:34.440 INFO:teuthology.orchestra.run.smithi162.stdout:(32/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.8 MB/s | 90 kB 00:00 2024-09-06T14:16:34.465 INFO:teuthology.orchestra.run.smithi162.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 710 kB/s | 36 kB 00:00 2024-09-06T14:16:34.490 INFO:teuthology.orchestra.run.smithi162.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-09-06T14:16:34.515 INFO:teuthology.orchestra.run.smithi162.stdout:(35/37): python3-webob-1.8.8-2.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-09-06T14:16:34.532 INFO:teuthology.orchestra.run.smithi162.stdout:(36/37): python3-zc-lockfile-2.0-10.el9.noarch. 479 kB/s | 20 kB 00:00 2024-09-06T14:16:34.565 INFO:teuthology.orchestra.run.smithi162.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.1 MB/s | 427 kB 00:00 2024-09-06T14:16:34.569 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-09-06T14:16:34.569 INFO:teuthology.orchestra.run.smithi162.stdout:Total 13 MB/s | 31 MB 00:02 2024-09-06T14:16:34.719 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-09-06T14:16:34.760 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-09-06T14:16:34.760 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-09-06T14:16:35.059 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-09-06T14:16:35.061 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-09-06T14:16:36.250 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-test-2:19.3.0-4704.g8b38c33c.el9.x86_64 4/4 2024-09-06T14:16:36.251 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-test-2:19.3.0-4704.g8b38c33c.el9.x86_64 1/4 2024-09-06T14:16:36.251 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libcephsqlite-2:19.3.0-4704.g8b38c33c.el9.x86_64 2/4 2024-09-06T14:16:36.251 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-06T14:16:36.404 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-09-06T14:16:36.477 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-06T14:16:36.526 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-06T14:16:36.553 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-06T14:16:36.554 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:16:36.554 INFO:teuthology.orchestra.run.smithi142.stdout:Installed: 2024-09-06T14:16:36.554 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-test-2:19.3.0-4704.g8b38c33c.el9.x86_64 2024-09-06T14:16:36.554 INFO:teuthology.orchestra.run.smithi142.stdout: libcephsqlite-2:19.3.0-4704.g8b38c33c.el9.x86_64 2024-09-06T14:16:36.554 INFO:teuthology.orchestra.run.smithi142.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-06T14:16:36.554 INFO:teuthology.orchestra.run.smithi142.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-06T14:16:36.554 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:16:36.555 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2024-09-06T14:16:36.562 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-06T14:16:36.634 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-06T14:16:36.695 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-06T14:16:36.728 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-06T14:16:36.821 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-06T14:16:36.884 DEBUG:teuthology.orchestra.run.smithi142:> sudo yum -y install ceph 2024-09-06T14:16:36.932 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-06T14:16:36.981 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-06T14:16:37.024 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-06T14:16:37.054 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-06T14:16:37.088 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-06T14:16:37.122 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-06T14:16:37.162 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-06T14:16:37.197 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-06T14:16:37.234 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-06T14:16:37.283 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-06T14:16:37.315 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-06T14:16:37.388 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-06T14:16:37.472 INFO:teuthology.orchestra.run.smithi142.stdout:Last metadata expiration check: 0:00:41 ago on Fri 06 Sep 2024 02:15:56 PM UTC. 2024-09-06T14:16:37.528 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-06T14:16:37.613 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2024-09-06T14:16:37.615 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-06T14:16:37.616 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-09-06T14:16:37.616 INFO:teuthology.orchestra.run.smithi142.stdout: Package Arch Version Repository Size 2024-09-06T14:16:37.616 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-09-06T14:16:37.616 INFO:teuthology.orchestra.run.smithi142.stdout:Installing: 2024-09-06T14:16:37.616 INFO:teuthology.orchestra.run.smithi142.stdout: ceph x86_64 2:19.3.0-4704.g8b38c33c.el9 ceph 6.6 k 2024-09-06T14:16:37.616 INFO:teuthology.orchestra.run.smithi142.stdout:Installing dependencies: 2024-09-06T14:16:37.616 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mds x86_64 2:19.3.0-4704.g8b38c33c.el9 ceph 2.4 M 2024-09-06T14:16:37.616 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr x86_64 2:19.3.0-4704.g8b38c33c.el9 ceph 1.5 M 2024-09-06T14:16:37.616 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr-modules-core noarch 2:19.3.0-4704.g8b38c33c.el9 ceph-noarch 291 k 2024-09-06T14:16:37.616 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mon x86_64 2:19.3.0-4704.g8b38c33c.el9 ceph 4.8 M 2024-09-06T14:16:37.617 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-osd x86_64 2:19.3.0-4704.g8b38c33c.el9 ceph 17 M 2024-09-06T14:16:37.617 INFO:teuthology.orchestra.run.smithi142.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-06T14:16:37.617 INFO:teuthology.orchestra.run.smithi142.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-06T14:16:37.617 INFO:teuthology.orchestra.run.smithi142.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-06T14:16:37.617 INFO:teuthology.orchestra.run.smithi142.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-06T14:16:37.617 INFO:teuthology.orchestra.run.smithi142.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-06T14:16:37.617 INFO:teuthology.orchestra.run.smithi142.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-06T14:16:37.617 INFO:teuthology.orchestra.run.smithi142.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-06T14:16:37.617 INFO:teuthology.orchestra.run.smithi142.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-06T14:16:37.617 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-06T14:16:37.617 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-06T14:16:37.617 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jaraco-collections 2024-09-06T14:16:37.617 INFO:teuthology.orchestra.run.smithi142.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-06T14:16:37.618 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-06T14:16:37.618 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-06T14:16:37.618 INFO:teuthology.orchestra.run.smithi142.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-06T14:16:37.618 INFO:teuthology.orchestra.run.smithi142.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-06T14:16:37.618 INFO:teuthology.orchestra.run.smithi142.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-06T14:16:37.618 INFO:teuthology.orchestra.run.smithi142.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-06T14:16:37.618 INFO:teuthology.orchestra.run.smithi142.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-06T14:16:37.618 INFO:teuthology.orchestra.run.smithi142.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-06T14:16:37.618 INFO:teuthology.orchestra.run.smithi142.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-06T14:16:37.618 INFO:teuthology.orchestra.run.smithi142.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-06T14:16:37.618 INFO:teuthology.orchestra.run.smithi142.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-06T14:16:37.618 INFO:teuthology.orchestra.run.smithi142.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-06T14:16:37.618 INFO:teuthology.orchestra.run.smithi142.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-06T14:16:37.619 INFO:teuthology.orchestra.run.smithi142.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-06T14:16:37.619 INFO:teuthology.orchestra.run.smithi142.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-06T14:16:37.619 INFO:teuthology.orchestra.run.smithi142.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-06T14:16:37.619 INFO:teuthology.orchestra.run.smithi142.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-06T14:16:37.619 INFO:teuthology.orchestra.run.smithi142.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-06T14:16:37.619 INFO:teuthology.orchestra.run.smithi142.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-06T14:16:37.619 INFO:teuthology.orchestra.run.smithi142.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-06T14:16:37.619 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:16:37.619 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction Summary 2024-09-06T14:16:37.619 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-09-06T14:16:37.619 INFO:teuthology.orchestra.run.smithi142.stdout:Install 37 Packages 2024-09-06T14:16:37.619 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:16:37.620 INFO:teuthology.orchestra.run.smithi142.stdout:Total download size: 31 M 2024-09-06T14:16:37.620 INFO:teuthology.orchestra.run.smithi142.stdout:Installed size: 109 M 2024-09-06T14:16:37.620 INFO:teuthology.orchestra.run.smithi142.stdout:Downloading Packages: 2024-09-06T14:16:37.676 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-06T14:16:37.715 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-06T14:16:37.773 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-06T14:16:37.828 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-06T14:16:38.017 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-06T14:16:38.069 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-06T14:16:38.156 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-06T14:16:38.232 INFO:teuthology.orchestra.run.smithi142.stdout:(1/37): ceph-19.3.0-4704.g8b38c33c.el9.x86_64.r 47 kB/s | 6.6 kB 00:00 2024-09-06T14:16:38.253 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-06T14:16:38.293 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-06T14:16:38.391 INFO:teuthology.orchestra.run.smithi142.stdout:(2/37): ceph-mgr-19.3.0-4704.g8b38c33c.el9.x86_ 5.1 MB/s | 1.5 MB 00:00 2024-09-06T14:16:38.422 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-06T14:16:38.466 INFO:teuthology.orchestra.run.smithi142.stdout:(3/37): ceph-mds-19.3.0-4704.g8b38c33c.el9.x86_ 6.4 MB/s | 2.4 MB 00:00 2024-09-06T14:16:38.566 INFO:teuthology.orchestra.run.smithi142.stdout:(4/37): ceph-mgr-modules-core-19.3.0-4704.g8b38 2.8 MB/s | 291 kB 00:00 2024-09-06T14:16:38.583 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-4704.g8b38c33c.el9. 32/37 2024-09-06T14:16:38.612 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mgr-2:19.3.0-4704.g8b38c33c.el9.x86_64 33/37 2024-09-06T14:16:38.633 INFO:teuthology.orchestra.run.smithi142.stdout:(5/37): ceph-mon-19.3.0-4704.g8b38c33c.el9.x86_ 12 MB/s | 4.8 MB 00:00 2024-09-06T14:16:38.640 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mgr-2:19.3.0-4704.g8b38c33c.el9.x86_64 33/37 2024-09-06T14:16:38.640 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-06T14:16:38.640 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-06T14:16:38.640 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-06T14:16:38.640 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-06T14:16:38.640 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-06T14:16:38.800 INFO:teuthology.orchestra.run.smithi142.stdout:(6/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.1 MB/s | 253 kB 00:00 2024-09-06T14:16:38.992 INFO:teuthology.orchestra.run.smithi142.stdout:(7/37): ceph-osd-19.3.0-4704.g8b38c33c.el9.x86_ 29 MB/s | 17 MB 00:00 2024-09-06T14:16:39.017 INFO:teuthology.orchestra.run.smithi142.stdout:(8/37): python3-ply-3.11-14.el9.noarch.rpm 491 kB/s | 106 kB 00:00 2024-09-06T14:16:39.093 INFO:teuthology.orchestra.run.smithi142.stdout:(9/37): python3-requests-2.25.1-8.el9.noarch.rp 1.6 MB/s | 125 kB 00:00 2024-09-06T14:16:39.151 INFO:teuthology.orchestra.run.smithi142.stdout:(10/37): python3-pycparser-2.20-6.el9.noarch.rp 851 kB/s | 135 kB 00:00 2024-09-06T14:16:39.185 INFO:teuthology.orchestra.run.smithi142.stdout:(11/37): python3-cryptography-36.0.1-4.el9.x86_ 2.2 MB/s | 1.2 MB 00:00 2024-09-06T14:16:39.210 INFO:teuthology.orchestra.run.smithi142.stdout:(12/37): python3-urllib3-1.26.5-6.el9.noarch.rp 1.8 MB/s | 215 kB 00:00 2024-09-06T14:16:39.277 INFO:teuthology.orchestra.run.smithi142.stdout:(13/37): python3-markupsafe-1.1.1-12.el9.x86_64 520 kB/s | 35 kB 00:00 2024-09-06T14:16:39.302 INFO:teuthology.orchestra.run.smithi142.stdout:(14/37): lua-5.4.4-4.el9.x86_64.rpm 1.2 MB/s | 188 kB 00:00 2024-09-06T14:16:39.327 INFO:teuthology.orchestra.run.smithi142.stdout:(15/37): python3-mako-1.1.4-6.el9.noarch.rpm 1.2 MB/s | 172 kB 00:00 2024-09-06T14:16:39.353 INFO:teuthology.orchestra.run.smithi142.stdout:(16/37): python3-packaging-20.9-5.el9.noarch.rp 1.0 MB/s | 77 kB 00:00 2024-09-06T14:16:39.370 INFO:teuthology.orchestra.run.smithi142.stdout:(17/37): python3-pytz-2021.1-5.el9.noarch.rpm 758 kB/s | 51 kB 00:00 2024-09-06T14:16:39.387 INFO:teuthology.orchestra.run.smithi142.stdout:(18/37): python3-toml-0.10.2-6.el9.noarch.rpm 1.2 MB/s | 42 kB 00:00 2024-09-06T14:16:39.446 INFO:teuthology.orchestra.run.smithi142.stdout:(19/37): luarocks-3.9.2-5.el9.noarch.rpm 2.0 MB/s | 151 kB 00:00 2024-09-06T14:16:39.487 INFO:teuthology.orchestra.run.smithi142.stdout:(20/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 433 kB/s | 43 kB 00:00 2024-09-06T14:16:39.513 INFO:teuthology.orchestra.run.smithi142.stdout:(21/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.5 MB/s | 172 kB 00:00 2024-09-06T14:16:39.530 INFO:teuthology.orchestra.run.smithi142.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 642 kB/s | 11 kB 00:00 2024-09-06T14:16:39.582 INFO:teuthology.orchestra.run.smithi142.stdout:(23/37): python3-jaraco-classes-3.2.1-5.el9.noa 338 kB/s | 18 kB 00:00 2024-09-06T14:16:39.680 INFO:teuthology.orchestra.run.smithi142.stdout:(24/37): python3-cherrypy-18.6.1-2.el9.noarch.r 1.8 MB/s | 358 kB 00:00 2024-09-06T14:16:39.705 INFO:teuthology.orchestra.run.smithi142.stdout:(25/37): python3-jaraco-collections-3.0.0-8.el9 188 kB/s | 23 kB 00:00 2024-09-06T14:16:39.723 INFO:teuthology.orchestra.run.smithi142.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 564 kB/s | 19 kB 00:00 2024-09-06T14:16:39.739 INFO:teuthology.orchestra.run.smithi142.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 585 kB/s | 20 kB 00:00 2024-09-06T14:16:39.756 INFO:teuthology.orchestra.run.smithi142.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 1.4 MB/s | 46 kB 00:00 2024-09-06T14:16:39.782 INFO:teuthology.orchestra.run.smithi142.stdout:(29/37): python3-more-itertools-8.12.0-2.el9.no 1.8 MB/s | 79 kB 00:00 2024-09-06T14:16:39.798 INFO:teuthology.orchestra.run.smithi142.stdout:(30/37): python3-portend-3.1.0-2.el9.noarch.rpm 1.0 MB/s | 16 kB 00:00 2024-09-06T14:16:39.832 INFO:teuthology.orchestra.run.smithi142.stdout:(31/37): python3-pecan-1.4.2-3.el9.noarch.rpm 3.5 MB/s | 272 kB 00:00 2024-09-06T14:16:39.857 INFO:teuthology.orchestra.run.smithi142.stdout:(32/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.5 MB/s | 90 kB 00:00 2024-09-06T14:16:39.874 INFO:teuthology.orchestra.run.smithi142.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 856 kB/s | 36 kB 00:00 2024-09-06T14:16:39.908 INFO:teuthology.orchestra.run.smithi142.stdout:(34/37): python3-webob-1.8.8-2.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-09-06T14:16:39.925 INFO:teuthology.orchestra.run.smithi142.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 1.2 MB/s | 20 kB 00:00 2024-09-06T14:16:39.950 INFO:teuthology.orchestra.run.smithi142.stdout:(36/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 5.5 MB/s | 427 kB 00:00 2024-09-06T14:16:40.017 INFO:teuthology.orchestra.run.smithi142.stdout:(37/37): lua-devel-5.4.4-4.el9.x86_64.rpm 34 kB/s | 22 kB 00:00 2024-09-06T14:16:40.022 INFO:teuthology.orchestra.run.smithi142.stdout:-------------------------------------------------------------------------------- 2024-09-06T14:16:40.022 INFO:teuthology.orchestra.run.smithi142.stdout:Total 13 MB/s | 31 MB 00:02 2024-09-06T14:16:40.105 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-osd-2:19.3.0-4704.g8b38c33c.el9.x86_64 34/37 2024-09-06T14:16:40.133 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-osd-2:19.3.0-4704.g8b38c33c.el9.x86_64 34/37 2024-09-06T14:16:40.133 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-06T14:16:40.133 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-06T14:16:40.133 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-06T14:16:40.133 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-06T14:16:40.133 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-06T14:16:40.168 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction check 2024-09-06T14:16:40.208 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction check succeeded. 2024-09-06T14:16:40.208 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction test 2024-09-06T14:16:40.507 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction test succeeded. 2024-09-06T14:16:40.508 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction 2024-09-06T14:16:40.590 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mon-2:19.3.0-4704.g8b38c33c.el9.x86_64 35/37 2024-09-06T14:16:40.635 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mon-2:19.3.0-4704.g8b38c33c.el9.x86_64 35/37 2024-09-06T14:16:40.635 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-06T14:16:40.635 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-06T14:16:40.635 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-06T14:16:40.636 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-06T14:16:40.636 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-06T14:16:40.840 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mds-2:19.3.0-4704.g8b38c33c.el9.x86_64 36/37 2024-09-06T14:16:40.863 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mds-2:19.3.0-4704.g8b38c33c.el9.x86_64 36/37 2024-09-06T14:16:40.864 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-06T14:16:40.864 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-06T14:16:40.864 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-06T14:16:40.864 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-06T14:16:40.864 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-06T14:16:40.926 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-2:19.3.0-4704.g8b38c33c.el9.x86_64 37/37 2024-09-06T14:16:41.814 INFO:teuthology.orchestra.run.smithi142.stdout: Preparing : 1/1 2024-09-06T14:16:41.886 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-06T14:16:41.920 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-06T14:16:41.947 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-06T14:16:42.018 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-06T14:16:42.063 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-06T14:16:42.096 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-06T14:16:42.197 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-06T14:16:42.274 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-06T14:16:42.315 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-06T14:16:42.350 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-06T14:16:42.380 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-06T14:16:42.422 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-06T14:16:42.464 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-06T14:16:42.505 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-06T14:16:42.539 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-06T14:16:42.574 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-06T14:16:42.594 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-2:19.3.0-4704.g8b38c33c.el9.x86_64 37/37 2024-09-06T14:16:42.594 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-2:19.3.0-4704.g8b38c33c.el9.x86_64 1/37 2024-09-06T14:16:42.594 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mds-2:19.3.0-4704.g8b38c33c.el9.x86_64 2/37 2024-09-06T14:16:42.594 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mgr-2:19.3.0-4704.g8b38c33c.el9.x86_64 3/37 2024-09-06T14:16:42.594 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mon-2:19.3.0-4704.g8b38c33c.el9.x86_64 4/37 2024-09-06T14:16:42.594 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-osd-2:19.3.0-4704.g8b38c33c.el9.x86_64 5/37 2024-09-06T14:16:42.594 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-4704.g8b38c33c.el9. 6/37 2024-09-06T14:16:42.594 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-06T14:16:42.594 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-06T14:16:42.594 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-06T14:16:42.595 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-06T14:16:42.595 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-06T14:16:42.595 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-06T14:16:42.595 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-06T14:16:42.595 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-06T14:16:42.595 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-06T14:16:42.595 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-06T14:16:42.595 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-06T14:16:42.595 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-06T14:16:42.595 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-06T14:16:42.595 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-06T14:16:42.595 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-06T14:16:42.595 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-06T14:16:42.595 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-06T14:16:42.596 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-06T14:16:42.596 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-06T14:16:42.596 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-06T14:16:42.596 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-06T14:16:42.596 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-06T14:16:42.596 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-06T14:16:42.596 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-06T14:16:42.596 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-06T14:16:42.596 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-06T14:16:42.596 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-06T14:16:42.596 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-06T14:16:42.596 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-06T14:16:42.596 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-06T14:16:42.623 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-06T14:16:42.679 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-06T14:16:42.753 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-06T14:16:42.893 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-06T14:16:42.972 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-06T14:16:43.033 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-06T14:16:43.080 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-06T14:16:43.137 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-06T14:16:43.201 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-06T14:16:43.389 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-06T14:16:43.442 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-06T14:16:43.521 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-06T14:16:43.543 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-06T14:16:43.543 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-06T14:16:43.543 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-09-06T14:16:43.543 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-2:19.3.0-4704.g8b38c33c.el9.x86_64 2024-09-06T14:16:43.543 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mds-2:19.3.0-4704.g8b38c33c.el9.x86_64 2024-09-06T14:16:43.544 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-2:19.3.0-4704.g8b38c33c.el9.x86_64 2024-09-06T14:16:43.544 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-modules-core-2:19.3.0-4704.g8b38c33c.el9.noarch 2024-09-06T14:16:43.544 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mon-2:19.3.0-4704.g8b38c33c.el9.x86_64 2024-09-06T14:16:43.544 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-osd-2:19.3.0-4704.g8b38c33c.el9.x86_64 2024-09-06T14:16:43.544 INFO:teuthology.orchestra.run.smithi162.stdout: lua-5.4.4-4.el9.x86_64 2024-09-06T14:16:43.544 INFO:teuthology.orchestra.run.smithi162.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-06T14:16:43.544 INFO:teuthology.orchestra.run.smithi162.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-06T14:16:43.544 INFO:teuthology.orchestra.run.smithi162.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-06T14:16:43.544 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-06T14:16:43.544 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-06T14:16:43.545 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-06T14:16:43.545 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-06T14:16:43.545 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-06T14:16:43.545 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-06T14:16:43.545 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-06T14:16:43.545 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-06T14:16:43.545 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-06T14:16:43.545 INFO:teuthology.orchestra.run.smithi162.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-06T14:16:43.545 INFO:teuthology.orchestra.run.smithi162.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-06T14:16:43.545 INFO:teuthology.orchestra.run.smithi162.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-06T14:16:43.545 INFO:teuthology.orchestra.run.smithi162.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-06T14:16:43.546 INFO:teuthology.orchestra.run.smithi162.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-06T14:16:43.546 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-06T14:16:43.546 INFO:teuthology.orchestra.run.smithi162.stdout: python3-ply-3.11-14.el9.noarch 2024-09-06T14:16:43.546 INFO:teuthology.orchestra.run.smithi162.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-06T14:16:43.546 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-06T14:16:43.546 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-06T14:16:43.546 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-06T14:16:43.546 INFO:teuthology.orchestra.run.smithi162.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-06T14:16:43.546 INFO:teuthology.orchestra.run.smithi162.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-06T14:16:43.546 INFO:teuthology.orchestra.run.smithi162.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-06T14:16:43.546 INFO:teuthology.orchestra.run.smithi162.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-06T14:16:43.546 INFO:teuthology.orchestra.run.smithi162.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-06T14:16:43.547 INFO:teuthology.orchestra.run.smithi162.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-06T14:16:43.547 INFO:teuthology.orchestra.run.smithi162.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-06T14:16:43.547 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-06T14:16:43.547 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-09-06T14:16:43.636 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-06T14:16:43.674 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-06T14:16:43.804 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-06T14:16:43.945 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-base 2024-09-06T14:16:43.980 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-4704.g8b38c33c.el9. 32/37 2024-09-06T14:16:44.010 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-mgr-2:19.3.0-4704.g8b38c33c.el9.x86_64 33/37 2024-09-06T14:16:44.036 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-mgr-2:19.3.0-4704.g8b38c33c.el9.x86_64 33/37 2024-09-06T14:16:44.036 INFO:teuthology.orchestra.run.smithi142.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-06T14:16:44.036 INFO:teuthology.orchestra.run.smithi142.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-06T14:16:44.036 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-06T14:16:44.037 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-06T14:16:44.037 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:16:44.508 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:00:52 ago on Fri 06 Sep 2024 02:15:52 PM UTC. 2024-09-06T14:16:44.597 INFO:teuthology.orchestra.run.smithi162.stdout:Package ceph-base-2:19.3.0-4704.g8b38c33c.el9.x86_64 is already installed. 2024-09-06T14:16:44.640 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-09-06T14:16:44.641 INFO:teuthology.orchestra.run.smithi162.stdout:Nothing to do. 2024-09-06T14:16:44.641 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-09-06T14:16:44.716 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install cephadm 2024-09-06T14:16:45.271 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:00:53 ago on Fri 06 Sep 2024 02:15:52 PM UTC. 2024-09-06T14:16:45.403 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-09-06T14:16:45.403 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-06T14:16:45.403 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repository Size 2024-09-06T14:16:45.404 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-06T14:16:45.404 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-09-06T14:16:45.404 INFO:teuthology.orchestra.run.smithi162.stdout: cephadm noarch 2:19.3.0-4704.g8b38c33c.el9 ceph-noarch 783 k 2024-09-06T14:16:45.404 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-06T14:16:45.404 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-09-06T14:16:45.404 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-06T14:16:45.404 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-09-06T14:16:45.404 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-06T14:16:45.404 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 783 k 2024-09-06T14:16:45.404 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 789 k 2024-09-06T14:16:45.404 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-09-06T14:16:45.510 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-osd-2:19.3.0-4704.g8b38c33c.el9.x86_64 34/37 2024-09-06T14:16:45.538 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-osd-2:19.3.0-4704.g8b38c33c.el9.x86_64 34/37 2024-09-06T14:16:45.538 INFO:teuthology.orchestra.run.smithi142.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-06T14:16:45.538 INFO:teuthology.orchestra.run.smithi142.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-06T14:16:45.538 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-06T14:16:45.538 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-06T14:16:45.538 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:16:45.664 INFO:teuthology.orchestra.run.smithi162.stdout:cephadm-19.3.0-4704.g8b38c33c.el9.noarch.rpm 3.0 MB/s | 783 kB 00:00 2024-09-06T14:16:45.664 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-09-06T14:16:45.665 INFO:teuthology.orchestra.run.smithi162.stdout:Total 2.9 MB/s | 783 kB 00:00 2024-09-06T14:16:45.665 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-09-06T14:16:45.670 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-09-06T14:16:45.670 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-09-06T14:16:45.676 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-09-06T14:16:45.676 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-09-06T14:16:45.777 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-09-06T14:16:45.995 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-mon-2:19.3.0-4704.g8b38c33c.el9.x86_64 35/37 2024-09-06T14:16:46.018 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-mon-2:19.3.0-4704.g8b38c33c.el9.x86_64 35/37 2024-09-06T14:16:46.019 INFO:teuthology.orchestra.run.smithi142.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-06T14:16:46.019 INFO:teuthology.orchestra.run.smithi142.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-06T14:16:46.019 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-06T14:16:46.019 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-06T14:16:46.019 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:16:46.217 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: cephadm-2:19.3.0-4704.g8b38c33c.el9.noarch 1/1 2024-09-06T14:16:46.245 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-mds-2:19.3.0-4704.g8b38c33c.el9.x86_64 36/37 2024-09-06T14:16:46.269 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-mds-2:19.3.0-4704.g8b38c33c.el9.x86_64 36/37 2024-09-06T14:16:46.269 INFO:teuthology.orchestra.run.smithi142.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-06T14:16:46.269 INFO:teuthology.orchestra.run.smithi142.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-06T14:16:46.269 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-06T14:16:46.269 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-06T14:16:46.269 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:16:46.292 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : cephadm-2:19.3.0-4704.g8b38c33c.el9.noarch 1/1 2024-09-06T14:16:46.332 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-2:19.3.0-4704.g8b38c33c.el9.x86_64 37/37 2024-09-06T14:16:46.658 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: cephadm-2:19.3.0-4704.g8b38c33c.el9.noarch 1/1 2024-09-06T14:16:46.894 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : cephadm-2:19.3.0-4704.g8b38c33c.el9.noarch 1/1 2024-09-06T14:16:46.894 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-06T14:16:46.894 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-09-06T14:16:46.894 INFO:teuthology.orchestra.run.smithi162.stdout: cephadm-2:19.3.0-4704.g8b38c33c.el9.noarch 2024-09-06T14:16:46.894 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-06T14:16:46.895 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-09-06T14:16:47.103 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-immutable-object-cache 2024-09-06T14:16:47.675 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:00:55 ago on Fri 06 Sep 2024 02:15:52 PM UTC. 2024-09-06T14:16:47.808 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-09-06T14:16:47.808 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-06T14:16:47.808 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repo Size 2024-09-06T14:16:47.808 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-06T14:16:47.808 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-09-06T14:16:47.809 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-4704.g8b38c33c.el9 ceph 145 k 2024-09-06T14:16:47.809 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-06T14:16:47.809 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-09-06T14:16:47.809 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-06T14:16:47.809 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-09-06T14:16:47.809 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-06T14:16:47.809 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 145 k 2024-09-06T14:16:47.810 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 432 k 2024-09-06T14:16:47.810 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-09-06T14:16:47.917 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-2:19.3.0-4704.g8b38c33c.el9.x86_64 37/37 2024-09-06T14:16:47.917 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-2:19.3.0-4704.g8b38c33c.el9.x86_64 1/37 2024-09-06T14:16:47.917 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-mds-2:19.3.0-4704.g8b38c33c.el9.x86_64 2/37 2024-09-06T14:16:47.917 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-mgr-2:19.3.0-4704.g8b38c33c.el9.x86_64 3/37 2024-09-06T14:16:47.917 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-mon-2:19.3.0-4704.g8b38c33c.el9.x86_64 4/37 2024-09-06T14:16:47.917 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-osd-2:19.3.0-4704.g8b38c33c.el9.x86_64 5/37 2024-09-06T14:16:47.918 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-4704.g8b38c33c.el9. 6/37 2024-09-06T14:16:47.918 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-06T14:16:47.918 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-06T14:16:47.918 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-06T14:16:47.918 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-06T14:16:47.918 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-06T14:16:47.918 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-06T14:16:47.918 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-06T14:16:47.918 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-06T14:16:47.918 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-06T14:16:47.918 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-06T14:16:47.918 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-06T14:16:47.918 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-06T14:16:47.918 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-06T14:16:47.919 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-06T14:16:47.919 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-06T14:16:47.919 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-06T14:16:47.919 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-06T14:16:47.919 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-06T14:16:47.919 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-06T14:16:47.919 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-06T14:16:47.919 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-06T14:16:47.919 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-06T14:16:47.919 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-06T14:16:47.919 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-06T14:16:47.919 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-06T14:16:47.919 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-06T14:16:47.919 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-06T14:16:47.919 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-06T14:16:47.920 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-06T14:16:47.920 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-06T14:16:48.022 INFO:teuthology.orchestra.run.smithi162.stdout:ceph-immutable-object-cache-19.3.0-4704.g8b38c3 686 kB/s | 145 kB 00:00 2024-09-06T14:16:48.022 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-09-06T14:16:48.023 INFO:teuthology.orchestra.run.smithi162.stdout:Total 680 kB/s | 145 kB 00:00 2024-09-06T14:16:48.023 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-09-06T14:16:48.030 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-09-06T14:16:48.030 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-09-06T14:16:48.089 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-09-06T14:16:48.089 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-09-06T14:16:48.228 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-09-06T14:16:48.297 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-4704.g8b38c33c. 1/1 2024-09-06T14:16:48.322 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-4704.g8b38c33c. 1/1 2024-09-06T14:16:48.322 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-06T14:16:48.322 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-06T14:16:48.323 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-06T14:16:48.666 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-06T14:16:48.666 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:16:48.666 INFO:teuthology.orchestra.run.smithi142.stdout:Installed: 2024-09-06T14:16:48.666 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-2:19.3.0-4704.g8b38c33c.el9.x86_64 2024-09-06T14:16:48.666 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mds-2:19.3.0-4704.g8b38c33c.el9.x86_64 2024-09-06T14:16:48.666 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr-2:19.3.0-4704.g8b38c33c.el9.x86_64 2024-09-06T14:16:48.666 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr-modules-core-2:19.3.0-4704.g8b38c33c.el9.noarch 2024-09-06T14:16:48.666 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mon-2:19.3.0-4704.g8b38c33c.el9.x86_64 2024-09-06T14:16:48.667 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-osd-2:19.3.0-4704.g8b38c33c.el9.x86_64 2024-09-06T14:16:48.667 INFO:teuthology.orchestra.run.smithi142.stdout: lua-5.4.4-4.el9.x86_64 2024-09-06T14:16:48.667 INFO:teuthology.orchestra.run.smithi142.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-06T14:16:48.667 INFO:teuthology.orchestra.run.smithi142.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-06T14:16:48.667 INFO:teuthology.orchestra.run.smithi142.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-06T14:16:48.667 INFO:teuthology.orchestra.run.smithi142.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-06T14:16:48.667 INFO:teuthology.orchestra.run.smithi142.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-06T14:16:48.667 INFO:teuthology.orchestra.run.smithi142.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-06T14:16:48.667 INFO:teuthology.orchestra.run.smithi142.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-06T14:16:48.667 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-06T14:16:48.667 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-06T14:16:48.667 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-06T14:16:48.667 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-06T14:16:48.667 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-06T14:16:48.668 INFO:teuthology.orchestra.run.smithi142.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-06T14:16:48.668 INFO:teuthology.orchestra.run.smithi142.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-06T14:16:48.668 INFO:teuthology.orchestra.run.smithi142.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-06T14:16:48.668 INFO:teuthology.orchestra.run.smithi142.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-06T14:16:48.668 INFO:teuthology.orchestra.run.smithi142.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-06T14:16:48.668 INFO:teuthology.orchestra.run.smithi142.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-06T14:16:48.668 INFO:teuthology.orchestra.run.smithi142.stdout: python3-ply-3.11-14.el9.noarch 2024-09-06T14:16:48.668 INFO:teuthology.orchestra.run.smithi142.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-06T14:16:48.668 INFO:teuthology.orchestra.run.smithi142.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-06T14:16:48.668 INFO:teuthology.orchestra.run.smithi142.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-06T14:16:48.668 INFO:teuthology.orchestra.run.smithi142.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-06T14:16:48.668 INFO:teuthology.orchestra.run.smithi142.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-06T14:16:48.668 INFO:teuthology.orchestra.run.smithi142.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-06T14:16:48.669 INFO:teuthology.orchestra.run.smithi142.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-06T14:16:48.669 INFO:teuthology.orchestra.run.smithi142.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-06T14:16:48.669 INFO:teuthology.orchestra.run.smithi142.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-06T14:16:48.669 INFO:teuthology.orchestra.run.smithi142.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-06T14:16:48.669 INFO:teuthology.orchestra.run.smithi142.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-06T14:16:48.669 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:16:48.669 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2024-09-06T14:16:49.018 DEBUG:teuthology.orchestra.run.smithi142:> sudo yum -y install ceph-base 2024-09-06T14:16:49.233 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-4704.g8b38c33c. 1/1 2024-09-06T14:16:49.234 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-06T14:16:49.234 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-09-06T14:16:49.234 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-immutable-object-cache-2:19.3.0-4704.g8b38c33c.el9.x86_64 2024-09-06T14:16:49.234 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-06T14:16:49.234 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-09-06T14:16:49.485 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-mgr 2024-09-06T14:16:49.571 INFO:teuthology.orchestra.run.smithi142.stdout:Last metadata expiration check: 0:00:53 ago on Fri 06 Sep 2024 02:15:56 PM UTC. 2024-09-06T14:16:49.661 INFO:teuthology.orchestra.run.smithi142.stdout:Package ceph-base-2:19.3.0-4704.g8b38c33c.el9.x86_64 is already installed. 2024-09-06T14:16:49.702 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2024-09-06T14:16:49.703 INFO:teuthology.orchestra.run.smithi142.stdout:Nothing to do. 2024-09-06T14:16:49.703 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2024-09-06T14:16:49.769 DEBUG:teuthology.orchestra.run.smithi142:> sudo yum -y install cephadm 2024-09-06T14:16:50.055 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:00:58 ago on Fri 06 Sep 2024 02:15:52 PM UTC. 2024-09-06T14:16:50.148 INFO:teuthology.orchestra.run.smithi162.stdout:Package ceph-mgr-2:19.3.0-4704.g8b38c33c.el9.x86_64 is already installed. 2024-09-06T14:16:50.192 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-09-06T14:16:50.193 INFO:teuthology.orchestra.run.smithi162.stdout:Nothing to do. 2024-09-06T14:16:50.193 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-09-06T14:16:50.264 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-mgr-dashboard 2024-09-06T14:16:50.328 INFO:teuthology.orchestra.run.smithi142.stdout:Last metadata expiration check: 0:00:54 ago on Fri 06 Sep 2024 02:15:56 PM UTC. 2024-09-06T14:16:50.458 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2024-09-06T14:16:50.459 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-09-06T14:16:50.459 INFO:teuthology.orchestra.run.smithi142.stdout: Package Arch Version Repository Size 2024-09-06T14:16:50.459 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-09-06T14:16:50.459 INFO:teuthology.orchestra.run.smithi142.stdout:Installing: 2024-09-06T14:16:50.460 INFO:teuthology.orchestra.run.smithi142.stdout: cephadm noarch 2:19.3.0-4704.g8b38c33c.el9 ceph-noarch 783 k 2024-09-06T14:16:50.460 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:16:50.460 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction Summary 2024-09-06T14:16:50.460 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-09-06T14:16:50.460 INFO:teuthology.orchestra.run.smithi142.stdout:Install 1 Package 2024-09-06T14:16:50.460 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:16:50.460 INFO:teuthology.orchestra.run.smithi142.stdout:Total download size: 783 k 2024-09-06T14:16:50.460 INFO:teuthology.orchestra.run.smithi142.stdout:Installed size: 789 k 2024-09-06T14:16:50.460 INFO:teuthology.orchestra.run.smithi142.stdout:Downloading Packages: 2024-09-06T14:16:50.713 INFO:teuthology.orchestra.run.smithi142.stdout:cephadm-19.3.0-4704.g8b38c33c.el9.noarch.rpm 3.0 MB/s | 783 kB 00:00 2024-09-06T14:16:50.713 INFO:teuthology.orchestra.run.smithi142.stdout:-------------------------------------------------------------------------------- 2024-09-06T14:16:50.713 INFO:teuthology.orchestra.run.smithi142.stdout:Total 3.0 MB/s | 783 kB 00:00 2024-09-06T14:16:50.714 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction check 2024-09-06T14:16:50.719 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction check succeeded. 2024-09-06T14:16:50.719 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction test 2024-09-06T14:16:50.725 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction test succeeded. 2024-09-06T14:16:50.725 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction 2024-09-06T14:16:50.797 INFO:teuthology.orchestra.run.smithi142.stdout: Preparing : 1/1 2024-09-06T14:16:50.819 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:00:58 ago on Fri 06 Sep 2024 02:15:52 PM UTC. 2024-09-06T14:16:50.951 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-09-06T14:16:50.952 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-06T14:16:50.952 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repository Size 2024-09-06T14:16:50.952 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-06T14:16:50.952 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-09-06T14:16:50.952 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-dashboard noarch 2:19.3.0-4704.g8b38c33c.el9 ceph-noarch 14 M 2024-09-06T14:16:50.952 INFO:teuthology.orchestra.run.smithi162.stdout:Installing dependencies: 2024-09-06T14:16:50.952 INFO:teuthology.orchestra.run.smithi162.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-06T14:16:50.952 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-grafana-dashboards noarch 2:19.3.0-4704.g8b38c33c.el9 ceph-noarch 34 k 2024-09-06T14:16:50.952 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-prometheus-alerts noarch 2:19.3.0-4704.g8b38c33c.el9 ceph-noarch 16 k 2024-09-06T14:16:50.952 INFO:teuthology.orchestra.run.smithi162.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-06T14:16:50.953 INFO:teuthology.orchestra.run.smithi162.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-06T14:16:50.953 INFO:teuthology.orchestra.run.smithi162.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-06T14:16:50.953 INFO:teuthology.orchestra.run.smithi162.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-06T14:16:50.953 INFO:teuthology.orchestra.run.smithi162.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-06T14:16:50.953 INFO:teuthology.orchestra.run.smithi162.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-06T14:16:50.953 INFO:teuthology.orchestra.run.smithi162.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-06T14:16:50.953 INFO:teuthology.orchestra.run.smithi162.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-06T14:16:50.953 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-06T14:16:50.953 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-09-06T14:16:50.953 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-06T14:16:50.953 INFO:teuthology.orchestra.run.smithi162.stdout:Install 12 Packages 2024-09-06T14:16:50.953 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-06T14:16:50.954 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 19 M 2024-09-06T14:16:50.954 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 159 M 2024-09-06T14:16:50.954 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-09-06T14:16:51.170 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: cephadm-2:19.3.0-4704.g8b38c33c.el9.noarch 1/1 2024-09-06T14:16:51.289 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : cephadm-2:19.3.0-4704.g8b38c33c.el9.noarch 1/1 2024-09-06T14:16:51.352 INFO:teuthology.orchestra.run.smithi162.stdout:(1/12): ceph-grafana-dashboards-19.3.0-4704.g8b 234 kB/s | 34 kB 00:00 2024-09-06T14:16:51.369 INFO:teuthology.orchestra.run.smithi162.stdout:(2/12): ceph-prometheus-alerts-19.3.0-4704.g8b3 101 kB/s | 16 kB 00:00 2024-09-06T14:16:51.650 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: cephadm-2:19.3.0-4704.g8b38c33c.el9.noarch 1/1 2024-09-06T14:16:51.853 INFO:teuthology.orchestra.run.smithi162.stdout:(3/12): ceph-mgr-dashboard-19.3.0-4704.g8b38c33 22 MB/s | 14 MB 00:00 2024-09-06T14:16:51.903 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : cephadm-2:19.3.0-4704.g8b38c33c.el9.noarch 1/1 2024-09-06T14:16:51.904 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:16:51.904 INFO:teuthology.orchestra.run.smithi142.stdout:Installed: 2024-09-06T14:16:51.904 INFO:teuthology.orchestra.run.smithi142.stdout: cephadm-2:19.3.0-4704.g8b38c33c.el9.noarch 2024-09-06T14:16:51.904 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:16:51.904 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2024-09-06T14:16:52.100 DEBUG:teuthology.orchestra.run.smithi142:> sudo yum -y install ceph-immutable-object-cache 2024-09-06T14:16:52.119 INFO:teuthology.orchestra.run.smithi162.stdout:(4/12): protobuf-compiler-3.14.0-13.el9.x86_64. 3.2 MB/s | 863 kB 00:00 2024-09-06T14:16:52.187 INFO:teuthology.orchestra.run.smithi162.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 8.0 MB/s | 548 kB 00:00 2024-09-06T14:16:52.212 INFO:teuthology.orchestra.run.smithi162.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 779 kB/s | 19 kB 00:00 2024-09-06T14:16:52.254 INFO:teuthology.orchestra.run.smithi162.stdout:(7/12): python3-protobuf-3.14.0-13.el9.noarch.r 304 kB/s | 269 kB 00:00 2024-09-06T14:16:52.313 INFO:teuthology.orchestra.run.smithi162.stdout:(8/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 20 MB/s | 2.0 MB 00:00 2024-09-06T14:16:52.338 INFO:teuthology.orchestra.run.smithi162.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 1.3 MB/s | 31 kB 00:00 2024-09-06T14:16:52.363 INFO:teuthology.orchestra.run.smithi162.stdout:(10/12): python3-grpcio-tools-1.46.7-10.el9.x86 1.3 MB/s | 144 kB 00:00 2024-09-06T14:16:52.388 INFO:teuthology.orchestra.run.smithi162.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 3.7 MB/s | 188 kB 00:00 2024-09-06T14:16:52.480 INFO:teuthology.orchestra.run.smithi162.stdout:(12/12): protobuf-3.14.0-13.el9.x86_64.rpm 916 kB/s | 1.0 MB 00:01 2024-09-06T14:16:52.486 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-09-06T14:16:52.486 INFO:teuthology.orchestra.run.smithi162.stdout:Total 13 MB/s | 19 MB 00:01 2024-09-06T14:16:52.536 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-09-06T14:16:52.556 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-09-06T14:16:52.556 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-09-06T14:16:52.676 INFO:teuthology.orchestra.run.smithi142.stdout:Last metadata expiration check: 0:00:56 ago on Fri 06 Sep 2024 02:15:56 PM UTC. 2024-09-06T14:16:52.780 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-09-06T14:16:52.781 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-09-06T14:16:52.813 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2024-09-06T14:16:52.813 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-09-06T14:16:52.813 INFO:teuthology.orchestra.run.smithi142.stdout: Package Arch Version Repo Size 2024-09-06T14:16:52.813 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-09-06T14:16:52.814 INFO:teuthology.orchestra.run.smithi142.stdout:Installing: 2024-09-06T14:16:52.814 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-4704.g8b38c33c.el9 ceph 145 k 2024-09-06T14:16:52.814 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:16:52.814 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction Summary 2024-09-06T14:16:52.814 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-09-06T14:16:52.814 INFO:teuthology.orchestra.run.smithi142.stdout:Install 1 Package 2024-09-06T14:16:52.814 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:16:52.815 INFO:teuthology.orchestra.run.smithi142.stdout:Total download size: 145 k 2024-09-06T14:16:52.815 INFO:teuthology.orchestra.run.smithi142.stdout:Installed size: 432 k 2024-09-06T14:16:52.815 INFO:teuthology.orchestra.run.smithi142.stdout:Downloading Packages: 2024-09-06T14:16:53.035 INFO:teuthology.orchestra.run.smithi142.stdout:ceph-immutable-object-cache-19.3.0-4704.g8b38c3 660 kB/s | 145 kB 00:00 2024-09-06T14:16:53.036 INFO:teuthology.orchestra.run.smithi142.stdout:-------------------------------------------------------------------------------- 2024-09-06T14:16:53.036 INFO:teuthology.orchestra.run.smithi142.stdout:Total 656 kB/s | 145 kB 00:00 2024-09-06T14:16:53.036 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction check 2024-09-06T14:16:53.043 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction check succeeded. 2024-09-06T14:16:53.043 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction test 2024-09-06T14:16:53.102 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction test succeeded. 2024-09-06T14:16:53.103 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction 2024-09-06T14:16:53.243 INFO:teuthology.orchestra.run.smithi142.stdout: Preparing : 1/1 2024-09-06T14:16:53.303 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-4704.g8b38c33c. 1/1 2024-09-06T14:16:53.328 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-4704.g8b38c33c. 1/1 2024-09-06T14:16:53.328 INFO:teuthology.orchestra.run.smithi142.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-06T14:16:53.328 INFO:teuthology.orchestra.run.smithi142.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-06T14:16:53.328 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:16:53.361 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-09-06T14:16:53.439 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-06T14:16:53.491 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-06T14:16:53.537 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-06T14:16:53.568 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-06T14:16:53.645 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-06T14:16:53.804 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-06T14:16:53.897 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-06T14:16:53.940 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-06T14:16:53.969 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-06T14:16:54.001 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-4704.g8b38c33c.el9 10/12 2024-09-06T14:16:54.203 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-4704.g8b38c33c. 1/1 2024-09-06T14:16:54.203 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:16:54.203 INFO:teuthology.orchestra.run.smithi142.stdout:Installed: 2024-09-06T14:16:54.204 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-immutable-object-cache-2:19.3.0-4704.g8b38c33c.el9.x86_64 2024-09-06T14:16:54.204 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:16:54.204 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2024-09-06T14:16:54.440 DEBUG:teuthology.orchestra.run.smithi142:> sudo yum -y install ceph-mgr 2024-09-06T14:16:54.999 INFO:teuthology.orchestra.run.smithi142.stdout:Last metadata expiration check: 0:00:59 ago on Fri 06 Sep 2024 02:15:56 PM UTC. 2024-09-06T14:16:55.092 INFO:teuthology.orchestra.run.smithi142.stdout:Package ceph-mgr-2:19.3.0-4704.g8b38c33c.el9.x86_64 is already installed. 2024-09-06T14:16:55.134 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2024-09-06T14:16:55.135 INFO:teuthology.orchestra.run.smithi142.stdout:Nothing to do. 2024-09-06T14:16:55.135 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2024-09-06T14:16:55.213 DEBUG:teuthology.orchestra.run.smithi142:> sudo yum -y install ceph-mgr-dashboard 2024-09-06T14:16:55.767 INFO:teuthology.orchestra.run.smithi142.stdout:Last metadata expiration check: 0:00:59 ago on Fri 06 Sep 2024 02:15:56 PM UTC. 2024-09-06T14:16:55.898 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2024-09-06T14:16:55.899 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-09-06T14:16:55.899 INFO:teuthology.orchestra.run.smithi142.stdout: Package Arch Version Repository Size 2024-09-06T14:16:55.900 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-09-06T14:16:55.900 INFO:teuthology.orchestra.run.smithi142.stdout:Installing: 2024-09-06T14:16:55.900 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr-dashboard noarch 2:19.3.0-4704.g8b38c33c.el9 ceph-noarch 14 M 2024-09-06T14:16:55.900 INFO:teuthology.orchestra.run.smithi142.stdout:Installing dependencies: 2024-09-06T14:16:55.900 INFO:teuthology.orchestra.run.smithi142.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-06T14:16:55.900 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-grafana-dashboards noarch 2:19.3.0-4704.g8b38c33c.el9 ceph-noarch 34 k 2024-09-06T14:16:55.900 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-prometheus-alerts noarch 2:19.3.0-4704.g8b38c33c.el9 ceph-noarch 16 k 2024-09-06T14:16:55.900 INFO:teuthology.orchestra.run.smithi142.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-06T14:16:55.900 INFO:teuthology.orchestra.run.smithi142.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-06T14:16:55.900 INFO:teuthology.orchestra.run.smithi142.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-06T14:16:55.900 INFO:teuthology.orchestra.run.smithi142.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-06T14:16:55.900 INFO:teuthology.orchestra.run.smithi142.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-06T14:16:55.900 INFO:teuthology.orchestra.run.smithi142.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-06T14:16:55.901 INFO:teuthology.orchestra.run.smithi142.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-06T14:16:55.901 INFO:teuthology.orchestra.run.smithi142.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-06T14:16:55.901 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:16:55.901 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction Summary 2024-09-06T14:16:55.901 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-09-06T14:16:55.901 INFO:teuthology.orchestra.run.smithi142.stdout:Install 12 Packages 2024-09-06T14:16:55.901 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:16:55.902 INFO:teuthology.orchestra.run.smithi142.stdout:Total download size: 19 M 2024-09-06T14:16:55.902 INFO:teuthology.orchestra.run.smithi142.stdout:Installed size: 159 M 2024-09-06T14:16:55.902 INFO:teuthology.orchestra.run.smithi142.stdout:Downloading Packages: 2024-09-06T14:16:56.646 INFO:teuthology.orchestra.run.smithi142.stdout:(1/12): ceph-grafana-dashboards-19.3.0-4704.g8b 224 kB/s | 34 kB 00:00 2024-09-06T14:16:56.671 INFO:teuthology.orchestra.run.smithi142.stdout:(2/12): ceph-prometheus-alerts-19.3.0-4704.g8b3 92 kB/s | 16 kB 00:00 2024-09-06T14:16:56.889 INFO:teuthology.orchestra.run.smithi142.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.2 MB/s | 269 kB 00:00 2024-09-06T14:16:57.005 INFO:teuthology.orchestra.run.smithi142.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 2.8 MB/s | 1.0 MB 00:00 2024-09-06T14:16:57.039 INFO:teuthology.orchestra.run.smithi142.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 5.6 MB/s | 863 kB 00:00 2024-09-06T14:16:57.205 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-4704.g8b38c33c.el 11/12 2024-09-06T14:16:57.215 INFO:teuthology.orchestra.run.smithi142.stdout:(6/12): ceph-mgr-dashboard-19.3.0-4704.g8b38c33 20 MB/s | 14 MB 00:00 2024-09-06T14:16:57.240 INFO:teuthology.orchestra.run.smithi142.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 98 kB/s | 19 kB 00:00 2024-09-06T14:16:57.265 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-4704.g8b38c33c.el9.noa 12/12 2024-09-06T14:16:57.274 INFO:teuthology.orchestra.run.smithi142.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 2.0 MB/s | 548 kB 00:00 2024-09-06T14:16:57.299 INFO:teuthology.orchestra.run.smithi142.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 1.3 MB/s | 31 kB 00:00 2024-09-06T14:16:57.324 INFO:teuthology.orchestra.run.smithi142.stdout:(10/12): python3-grpcio-tools-1.46.7-10.el9.x86 1.8 MB/s | 144 kB 00:00 2024-09-06T14:16:57.349 INFO:teuthology.orchestra.run.smithi142.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 3.7 MB/s | 188 kB 00:00 2024-09-06T14:16:57.483 INFO:teuthology.orchestra.run.smithi142.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 7.6 MB/s | 2.0 MB 00:00 2024-09-06T14:16:57.487 INFO:teuthology.orchestra.run.smithi142.stdout:-------------------------------------------------------------------------------- 2024-09-06T14:16:57.487 INFO:teuthology.orchestra.run.smithi142.stdout:Total 12 MB/s | 19 MB 00:01 2024-09-06T14:16:57.536 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction check 2024-09-06T14:16:57.555 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction check succeeded. 2024-09-06T14:16:57.555 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction test 2024-09-06T14:16:57.780 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction test succeeded. 2024-09-06T14:16:57.780 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction 2024-09-06T14:16:58.366 INFO:teuthology.orchestra.run.smithi142.stdout: Preparing : 1/1 2024-09-06T14:16:58.441 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-06T14:16:58.485 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-06T14:16:58.540 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-06T14:16:58.579 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-06T14:16:58.654 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-06T14:16:58.779 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-06T14:16:58.866 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-06T14:16:58.908 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-06T14:16:58.945 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-06T14:16:58.986 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-4704.g8b38c33c.el9 10/12 2024-09-06T14:16:59.234 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-4704.g8b38c33c.el9.noa 12/12 2024-09-06T14:16:59.234 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-4704.g8b38c33c.el 1/12 2024-09-06T14:16:59.235 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-4704.g8b38c33c.el9.noa 2/12 2024-09-06T14:16:59.235 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-4704.g8b38c33c.el9 3/12 2024-09-06T14:16:59.235 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-06T14:16:59.235 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-06T14:16:59.235 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-06T14:16:59.235 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-06T14:16:59.236 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-06T14:16:59.236 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-06T14:16:59.236 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-06T14:16:59.236 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-06T14:16:59.573 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-06T14:16:59.574 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-06T14:16:59.574 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-09-06T14:16:59.574 INFO:teuthology.orchestra.run.smithi162.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-06T14:16:59.574 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-grafana-dashboards-2:19.3.0-4704.g8b38c33c.el9.noarch 2024-09-06T14:16:59.574 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-dashboard-2:19.3.0-4704.g8b38c33c.el9.noarch 2024-09-06T14:16:59.574 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-prometheus-alerts-2:19.3.0-4704.g8b38c33c.el9.noarch 2024-09-06T14:16:59.574 INFO:teuthology.orchestra.run.smithi162.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-06T14:16:59.574 INFO:teuthology.orchestra.run.smithi162.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-06T14:16:59.574 INFO:teuthology.orchestra.run.smithi162.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-06T14:16:59.574 INFO:teuthology.orchestra.run.smithi162.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-06T14:16:59.574 INFO:teuthology.orchestra.run.smithi162.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-06T14:16:59.574 INFO:teuthology.orchestra.run.smithi162.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-06T14:16:59.575 INFO:teuthology.orchestra.run.smithi162.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-06T14:16:59.575 INFO:teuthology.orchestra.run.smithi162.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-06T14:16:59.575 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-06T14:16:59.575 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-09-06T14:16:59.846 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-06T14:17:00.399 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:08 ago on Fri 06 Sep 2024 02:15:52 PM UTC. 2024-09-06T14:17:00.534 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-09-06T14:17:00.535 INFO:teuthology.orchestra.run.smithi162.stdout:======================================================================================= 2024-09-06T14:17:00.535 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repository Size 2024-09-06T14:17:00.535 INFO:teuthology.orchestra.run.smithi162.stdout:======================================================================================= 2024-09-06T14:17:00.535 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-09-06T14:17:00.535 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-4704.g8b38c33c.el9 ceph-noarch 7.4 M 2024-09-06T14:17:00.535 INFO:teuthology.orchestra.run.smithi162.stdout:Installing dependencies: 2024-09-06T14:17:00.535 INFO:teuthology.orchestra.run.smithi162.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-06T14:17:00.535 INFO:teuthology.orchestra.run.smithi162.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-06T14:17:00.535 INFO:teuthology.orchestra.run.smithi162.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-06T14:17:00.535 INFO:teuthology.orchestra.run.smithi162.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-06T14:17:00.536 INFO:teuthology.orchestra.run.smithi162.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-06T14:17:00.536 INFO:teuthology.orchestra.run.smithi162.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-06T14:17:00.536 INFO:teuthology.orchestra.run.smithi162.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-06T14:17:00.536 INFO:teuthology.orchestra.run.smithi162.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-06T14:17:00.536 INFO:teuthology.orchestra.run.smithi162.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-06T14:17:00.536 INFO:teuthology.orchestra.run.smithi162.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-06T14:17:00.536 INFO:teuthology.orchestra.run.smithi162.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-06T14:17:00.536 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-06T14:17:00.536 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-09-06T14:17:00.536 INFO:teuthology.orchestra.run.smithi162.stdout:======================================================================================= 2024-09-06T14:17:00.536 INFO:teuthology.orchestra.run.smithi162.stdout:Install 12 Packages 2024-09-06T14:17:00.536 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-06T14:17:00.537 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 38 M 2024-09-06T14:17:00.537 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 212 M 2024-09-06T14:17:00.537 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-09-06T14:17:00.798 INFO:teuthology.orchestra.run.smithi162.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 870 kB/s | 191 kB 00:00 2024-09-06T14:17:00.873 INFO:teuthology.orchestra.run.smithi162.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 2.7 MB/s | 801 kB 00:00 2024-09-06T14:17:00.898 INFO:teuthology.orchestra.run.smithi162.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 327 kB/s | 33 kB 00:00 2024-09-06T14:17:00.940 INFO:teuthology.orchestra.run.smithi162.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 435 kB/s | 18 kB 00:00 2024-09-06T14:17:00.990 INFO:teuthology.orchestra.run.smithi162.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 772 kB/s | 38 kB 00:00 2024-09-06T14:17:01.099 INFO:teuthology.orchestra.run.smithi162.stdout:(6/12): ceph-mgr-diskprediction-local-19.3.0-47 14 MB/s | 7.4 MB 00:00 2024-09-06T14:17:01.233 INFO:teuthology.orchestra.run.smithi162.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 8.3 MB/s | 3.0 MB 00:00 2024-09-06T14:17:01.316 INFO:teuthology.orchestra.run.smithi162.stdout:(8/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 15 MB/s | 4.9 MB 00:00 2024-09-06T14:17:01.341 INFO:teuthology.orchestra.run.smithi162.stdout:(9/12): python3-devel-3.9.19-8.el9.x86_64.rpm 1.0 MB/s | 245 kB 00:00 2024-09-06T14:17:01.367 INFO:teuthology.orchestra.run.smithi162.stdout:(10/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 4.9 MB/s | 250 kB 00:00 2024-09-06T14:17:01.459 INFO:teuthology.orchestra.run.smithi162.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 23 MB/s | 5.1 MB 00:00 2024-09-06T14:17:01.709 INFO:teuthology.orchestra.run.smithi162.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 44 MB/s | 16 MB 00:00 2024-09-06T14:17:01.710 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-09-06T14:17:01.710 INFO:teuthology.orchestra.run.smithi162.stdout:Total 33 MB/s | 38 MB 00:01 2024-09-06T14:17:02.060 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-09-06T14:17:02.085 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-09-06T14:17:02.085 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-09-06T14:17:02.184 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-4704.g8b38c33c.el 11/12 2024-09-06T14:17:02.240 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-4704.g8b38c33c.el9.noa 12/12 2024-09-06T14:17:02.497 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-09-06T14:17:02.497 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-09-06T14:17:03.258 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-09-06T14:17:03.322 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-06T14:17:03.357 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-06T14:17:03.456 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-06T14:17:03.500 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-06T14:17:03.701 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-06T14:17:03.733 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-06T14:17:03.810 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-06T14:17:04.191 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-4704.g8b38c33c.el9.noa 12/12 2024-09-06T14:17:04.191 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-4704.g8b38c33c.el 1/12 2024-09-06T14:17:04.191 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-4704.g8b38c33c.el9.noa 2/12 2024-09-06T14:17:04.191 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-4704.g8b38c33c.el9 3/12 2024-09-06T14:17:04.191 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-06T14:17:04.191 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-06T14:17:04.191 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-06T14:17:04.191 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-06T14:17:04.192 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-06T14:17:04.192 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-06T14:17:04.192 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-06T14:17:04.192 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-06T14:17:04.460 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-06T14:17:04.509 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-06T14:17:04.509 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:17:04.509 INFO:teuthology.orchestra.run.smithi142.stdout:Installed: 2024-09-06T14:17:04.509 INFO:teuthology.orchestra.run.smithi142.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-06T14:17:04.509 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-grafana-dashboards-2:19.3.0-4704.g8b38c33c.el9.noarch 2024-09-06T14:17:04.509 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr-dashboard-2:19.3.0-4704.g8b38c33c.el9.noarch 2024-09-06T14:17:04.509 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-prometheus-alerts-2:19.3.0-4704.g8b38c33c.el9.noarch 2024-09-06T14:17:04.509 INFO:teuthology.orchestra.run.smithi142.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-06T14:17:04.510 INFO:teuthology.orchestra.run.smithi142.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-06T14:17:04.510 INFO:teuthology.orchestra.run.smithi142.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-06T14:17:04.510 INFO:teuthology.orchestra.run.smithi142.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-06T14:17:04.510 INFO:teuthology.orchestra.run.smithi142.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-06T14:17:04.510 INFO:teuthology.orchestra.run.smithi142.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-06T14:17:04.510 INFO:teuthology.orchestra.run.smithi142.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-06T14:17:04.510 INFO:teuthology.orchestra.run.smithi142.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-06T14:17:04.510 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:17:04.510 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2024-09-06T14:17:04.579 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-06T14:17:04.768 DEBUG:teuthology.orchestra.run.smithi142:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-06T14:17:05.326 INFO:teuthology.orchestra.run.smithi142.stdout:Last metadata expiration check: 0:01:09 ago on Fri 06 Sep 2024 02:15:56 PM UTC. 2024-09-06T14:17:05.460 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2024-09-06T14:17:05.461 INFO:teuthology.orchestra.run.smithi142.stdout:======================================================================================= 2024-09-06T14:17:05.461 INFO:teuthology.orchestra.run.smithi142.stdout: Package Arch Version Repository Size 2024-09-06T14:17:05.461 INFO:teuthology.orchestra.run.smithi142.stdout:======================================================================================= 2024-09-06T14:17:05.461 INFO:teuthology.orchestra.run.smithi142.stdout:Installing: 2024-09-06T14:17:05.461 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-4704.g8b38c33c.el9 ceph-noarch 7.4 M 2024-09-06T14:17:05.462 INFO:teuthology.orchestra.run.smithi142.stdout:Installing dependencies: 2024-09-06T14:17:05.462 INFO:teuthology.orchestra.run.smithi142.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-06T14:17:05.462 INFO:teuthology.orchestra.run.smithi142.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-06T14:17:05.462 INFO:teuthology.orchestra.run.smithi142.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-06T14:17:05.462 INFO:teuthology.orchestra.run.smithi142.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-06T14:17:05.462 INFO:teuthology.orchestra.run.smithi142.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-06T14:17:05.462 INFO:teuthology.orchestra.run.smithi142.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-06T14:17:05.462 INFO:teuthology.orchestra.run.smithi142.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-06T14:17:05.462 INFO:teuthology.orchestra.run.smithi142.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-06T14:17:05.462 INFO:teuthology.orchestra.run.smithi142.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-06T14:17:05.462 INFO:teuthology.orchestra.run.smithi142.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-06T14:17:05.462 INFO:teuthology.orchestra.run.smithi142.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-06T14:17:05.462 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:17:05.463 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction Summary 2024-09-06T14:17:05.463 INFO:teuthology.orchestra.run.smithi142.stdout:======================================================================================= 2024-09-06T14:17:05.463 INFO:teuthology.orchestra.run.smithi142.stdout:Install 12 Packages 2024-09-06T14:17:05.463 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:17:05.463 INFO:teuthology.orchestra.run.smithi142.stdout:Total download size: 38 M 2024-09-06T14:17:05.463 INFO:teuthology.orchestra.run.smithi142.stdout:Installed size: 212 M 2024-09-06T14:17:05.463 INFO:teuthology.orchestra.run.smithi142.stdout:Downloading Packages: 2024-09-06T14:17:05.697 INFO:teuthology.orchestra.run.smithi142.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.3 MB/s | 191 kB 00:00 2024-09-06T14:17:05.773 INFO:teuthology.orchestra.run.smithi142.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 3.6 MB/s | 801 kB 00:00 2024-09-06T14:17:05.806 INFO:teuthology.orchestra.run.smithi142.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 300 kB/s | 33 kB 00:00 2024-09-06T14:17:05.825 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-06T14:17:05.856 INFO:teuthology.orchestra.run.smithi142.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 363 kB/s | 18 kB 00:00 2024-09-06T14:17:05.907 INFO:teuthology.orchestra.run.smithi142.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 770 kB/s | 38 kB 00:00 2024-09-06T14:17:06.032 INFO:teuthology.orchestra.run.smithi142.stdout:(6/12): ceph-mgr-diskprediction-local-19.3.0-47 16 MB/s | 7.4 MB 00:00 2024-09-06T14:17:06.224 INFO:teuthology.orchestra.run.smithi142.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 1.2 MB/s | 245 kB 00:00 2024-09-06T14:17:06.732 INFO:teuthology.orchestra.run.smithi142.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 3.1 MB/s | 3.0 MB 00:00 2024-09-06T14:17:06.816 INFO:teuthology.orchestra.run.smithi142.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 3.0 MB/s | 250 kB 00:00 2024-09-06T14:17:06.857 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-06T14:17:06.892 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-4704.g8b38c 12/12 2024-09-06T14:17:07.624 INFO:teuthology.orchestra.run.smithi142.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 3.6 MB/s | 5.1 MB 00:01 2024-09-06T14:17:07.699 INFO:teuthology.orchestra.run.smithi142.stdout:(11/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 2.8 MB/s | 4.9 MB 00:01 2024-09-06T14:17:09.466 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-4704.g8b38c 12/12 2024-09-06T14:17:09.466 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-4704.g8b38c 1/12 2024-09-06T14:17:09.466 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-06T14:17:09.466 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-06T14:17:09.466 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-06T14:17:09.466 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-06T14:17:09.466 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-06T14:17:09.466 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-06T14:17:09.467 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-06T14:17:09.467 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-06T14:17:09.467 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-06T14:17:09.467 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-06T14:17:09.857 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-06T14:17:09.857 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-06T14:17:09.857 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-09-06T14:17:09.857 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-diskprediction-local-2:19.3.0-4704.g8b38c33c.el9.noarch 2024-09-06T14:17:09.857 INFO:teuthology.orchestra.run.smithi162.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-06T14:17:09.857 INFO:teuthology.orchestra.run.smithi162.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-06T14:17:09.858 INFO:teuthology.orchestra.run.smithi162.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-06T14:17:09.858 INFO:teuthology.orchestra.run.smithi162.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-06T14:17:09.858 INFO:teuthology.orchestra.run.smithi162.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-06T14:17:09.858 INFO:teuthology.orchestra.run.smithi162.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-06T14:17:09.858 INFO:teuthology.orchestra.run.smithi162.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-06T14:17:09.858 INFO:teuthology.orchestra.run.smithi162.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-06T14:17:09.858 INFO:teuthology.orchestra.run.smithi162.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-06T14:17:09.858 INFO:teuthology.orchestra.run.smithi162.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-06T14:17:09.858 INFO:teuthology.orchestra.run.smithi162.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-06T14:17:09.858 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-06T14:17:09.858 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-09-06T14:17:10.215 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-mgr-rook 2024-09-06T14:17:10.790 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:18 ago on Fri 06 Sep 2024 02:15:52 PM UTC. 2024-09-06T14:17:10.923 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-09-06T14:17:10.924 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-06T14:17:10.924 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repository Size 2024-09-06T14:17:10.924 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-06T14:17:10.924 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-09-06T14:17:10.924 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-rook noarch 2:19.3.0-4704.g8b38c33c.el9 ceph-noarch 49 k 2024-09-06T14:17:10.924 INFO:teuthology.orchestra.run.smithi162.stdout:Installing dependencies: 2024-09-06T14:17:10.924 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-06T14:17:10.925 INFO:teuthology.orchestra.run.smithi162.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-06T14:17:10.925 INFO:teuthology.orchestra.run.smithi162.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-06T14:17:10.925 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-06T14:17:10.925 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-06T14:17:10.925 INFO:teuthology.orchestra.run.smithi162.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-06T14:17:10.925 INFO:teuthology.orchestra.run.smithi162.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-06T14:17:10.925 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-06T14:17:10.925 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-06T14:17:10.925 INFO:teuthology.orchestra.run.smithi162.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-06T14:17:10.925 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-06T14:17:10.925 INFO:teuthology.orchestra.run.smithi162.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-06T14:17:10.925 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-06T14:17:10.925 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-09-06T14:17:10.926 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-06T14:17:10.926 INFO:teuthology.orchestra.run.smithi162.stdout:Install 13 Packages 2024-09-06T14:17:10.926 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-06T14:17:10.926 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 2.2 M 2024-09-06T14:17:10.926 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 26 M 2024-09-06T14:17:10.926 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-09-06T14:17:11.379 INFO:teuthology.orchestra.run.smithi162.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 294 kB/s | 26 kB 00:00 2024-09-06T14:17:11.404 INFO:teuthology.orchestra.run.smithi162.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 170 kB/s | 19 kB 00:00 2024-09-06T14:17:11.463 INFO:teuthology.orchestra.run.smithi162.stdout:(3/13): ceph-mgr-rook-19.3.0-4704.g8b38c33c.el9 288 kB/s | 49 kB 00:00 2024-09-06T14:17:11.513 INFO:teuthology.orchestra.run.smithi162.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.6 MB/s | 222 kB 00:00 2024-09-06T14:17:11.538 INFO:teuthology.orchestra.run.smithi162.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.2 MB/s | 159 kB 00:00 2024-09-06T14:17:11.564 INFO:teuthology.orchestra.run.smithi162.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 54 kB 00:00 2024-09-06T14:17:11.622 INFO:teuthology.orchestra.run.smithi162.stdout:(7/13): python3-certifi-2023.05.07-4.el9.noarch 273 kB/s | 14 kB 00:00 2024-09-06T14:17:11.648 INFO:teuthology.orchestra.run.smithi162.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 296 kB/s | 32 kB 00:00 2024-09-06T14:17:11.690 INFO:teuthology.orchestra.run.smithi162.stdout:(9/13): python3-google-auth-2.34.0-1.el9.noarch 3.2 MB/s | 220 kB 00:00 2024-09-06T14:17:11.715 INFO:teuthology.orchestra.run.smithi162.stdout:(10/13): python3-pyasn1-modules-0.4.8-6.el9.noa 1.1 MB/s | 279 kB 00:00 2024-09-06T14:17:11.740 INFO:teuthology.orchestra.run.smithi162.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 1.2 MB/s | 59 kB 00:00 2024-09-06T14:17:11.790 INFO:teuthology.orchestra.run.smithi162.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.2 MB/s | 90 kB 00:00 2024-09-06T14:17:11.824 INFO:teuthology.orchestra.run.smithi162.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 5.8 MB/s | 1.0 MB 00:00 2024-09-06T14:17:11.827 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-09-06T14:17:11.827 INFO:teuthology.orchestra.run.smithi162.stdout:Total 2.5 MB/s | 2.2 MB 00:00 2024-09-06T14:17:11.891 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-09-06T14:17:11.906 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-09-06T14:17:11.906 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-09-06T14:17:11.992 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-09-06T14:17:11.994 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-09-06T14:17:12.249 INFO:teuthology.orchestra.run.smithi142.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 3.0 MB/s | 16 MB 00:05 2024-09-06T14:17:12.250 INFO:teuthology.orchestra.run.smithi142.stdout:-------------------------------------------------------------------------------- 2024-09-06T14:17:12.250 INFO:teuthology.orchestra.run.smithi142.stdout:Total 5.6 MB/s | 38 MB 00:06 2024-09-06T14:17:12.498 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-09-06T14:17:12.605 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction check 2024-09-06T14:17:12.632 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction check succeeded. 2024-09-06T14:17:12.632 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction test 2024-09-06T14:17:12.666 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-06T14:17:12.718 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-06T14:17:12.783 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-06T14:17:12.827 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-06T14:17:12.868 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-06T14:17:12.972 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-06T14:17:13.047 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction test succeeded. 2024-09-06T14:17:13.047 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction 2024-09-06T14:17:13.090 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-06T14:17:13.146 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-06T14:17:13.806 INFO:teuthology.orchestra.run.smithi142.stdout: Preparing : 1/1 2024-09-06T14:17:13.875 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-06T14:17:13.878 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-06T14:17:13.930 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-06T14:17:13.934 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-06T14:17:13.968 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-06T14:17:14.017 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-06T14:17:14.033 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-06T14:17:14.055 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mgr-rook-2:19.3.0-4704.g8b38c33c.el9.noarch 13/13 2024-09-06T14:17:14.073 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-06T14:17:14.258 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-06T14:17:14.306 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-06T14:17:14.390 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-06T14:17:14.809 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-4704.g8b38c33c.el9.noarch 13/13 2024-09-06T14:17:14.809 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mgr-rook-2:19.3.0-4704.g8b38c33c.el9.noarch 1/13 2024-09-06T14:17:14.809 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-06T14:17:14.809 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-06T14:17:14.809 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-06T14:17:14.809 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-06T14:17:14.809 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-06T14:17:14.809 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-06T14:17:14.809 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-06T14:17:14.810 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-06T14:17:14.810 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-06T14:17:14.810 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-06T14:17:14.810 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-06T14:17:15.029 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-06T14:17:15.135 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-06T14:17:15.152 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-06T14:17:15.152 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-06T14:17:15.153 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-09-06T14:17:15.153 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-rook-2:19.3.0-4704.g8b38c33c.el9.noarch 2024-09-06T14:17:15.153 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-06T14:17:15.153 INFO:teuthology.orchestra.run.smithi162.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-06T14:17:15.153 INFO:teuthology.orchestra.run.smithi162.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-06T14:17:15.153 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-06T14:17:15.153 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-06T14:17:15.153 INFO:teuthology.orchestra.run.smithi162.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-06T14:17:15.153 INFO:teuthology.orchestra.run.smithi162.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-06T14:17:15.153 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-06T14:17:15.153 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-06T14:17:15.153 INFO:teuthology.orchestra.run.smithi162.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-06T14:17:15.153 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-06T14:17:15.154 INFO:teuthology.orchestra.run.smithi162.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-06T14:17:15.154 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-06T14:17:15.154 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-09-06T14:17:15.393 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-mgr-cephadm 2024-09-06T14:17:15.952 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:23 ago on Fri 06 Sep 2024 02:15:52 PM UTC. 2024-09-06T14:17:16.087 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-09-06T14:17:16.087 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-06T14:17:16.087 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repository Size 2024-09-06T14:17:16.087 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-06T14:17:16.087 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-09-06T14:17:16.088 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-cephadm noarch 2:19.3.0-4704.g8b38c33c.el9 ceph-noarch 154 k 2024-09-06T14:17:16.088 INFO:teuthology.orchestra.run.smithi162.stdout:Installing dependencies: 2024-09-06T14:17:16.088 INFO:teuthology.orchestra.run.smithi162.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-06T14:17:16.088 INFO:teuthology.orchestra.run.smithi162.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-06T14:17:16.088 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-06T14:17:16.088 INFO:teuthology.orchestra.run.smithi162.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-06T14:17:16.088 INFO:teuthology.orchestra.run.smithi162.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-06T14:17:16.088 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-06T14:17:16.088 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-09-06T14:17:16.088 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-06T14:17:16.088 INFO:teuthology.orchestra.run.smithi162.stdout:Install 6 Packages 2024-09-06T14:17:16.088 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-06T14:17:16.089 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 7.0 M 2024-09-06T14:17:16.089 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 33 M 2024-09-06T14:17:16.089 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-09-06T14:17:16.371 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-06T14:17:16.629 INFO:teuthology.orchestra.run.smithi162.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.1 MB/s | 248 kB 00:00 2024-09-06T14:17:16.654 INFO:teuthology.orchestra.run.smithi162.stdout:(2/6): ceph-mgr-cephadm-19.3.0-4704.g8b38c33c.e 616 kB/s | 154 kB 00:00 2024-09-06T14:17:16.712 INFO:teuthology.orchestra.run.smithi162.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 989 kB/s | 58 kB 00:00 2024-09-06T14:17:16.746 INFO:teuthology.orchestra.run.smithi162.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 2.2 MB/s | 74 kB 00:00 2024-09-06T14:17:16.780 INFO:teuthology.orchestra.run.smithi162.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 3.6 MB/s | 548 kB 00:00 2024-09-06T14:17:17.138 INFO:teuthology.orchestra.run.smithi162.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 8.1 MB/s | 6.0 MB 00:00 2024-09-06T14:17:17.141 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-09-06T14:17:17.141 INFO:teuthology.orchestra.run.smithi162.stdout:Total 6.7 MB/s | 7.0 MB 00:01 2024-09-06T14:17:17.212 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-09-06T14:17:17.224 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-09-06T14:17:17.224 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-09-06T14:17:17.322 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-09-06T14:17:17.323 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-09-06T14:17:17.420 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-06T14:17:17.455 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-4704.g8b38c 12/12 2024-09-06T14:17:17.640 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-09-06T14:17:17.825 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-06T14:17:17.882 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-06T14:17:18.416 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-06T14:17:18.503 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-06T14:17:18.590 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-06T14:17:18.620 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-4704.g8b38c33c.el9.noarch 6/6 2024-09-06T14:17:19.446 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-4704.g8b38c33c.el9.noarch 6/6 2024-09-06T14:17:19.446 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-4704.g8b38c33c.el9.noarch 1/6 2024-09-06T14:17:19.446 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-06T14:17:19.446 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-06T14:17:19.446 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-06T14:17:19.446 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-06T14:17:19.719 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-06T14:17:19.720 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-06T14:17:19.720 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-09-06T14:17:19.720 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-cephadm-2:19.3.0-4704.g8b38c33c.el9.noarch 2024-09-06T14:17:19.720 INFO:teuthology.orchestra.run.smithi162.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-06T14:17:19.720 INFO:teuthology.orchestra.run.smithi162.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-06T14:17:19.720 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-06T14:17:19.720 INFO:teuthology.orchestra.run.smithi162.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-06T14:17:19.720 INFO:teuthology.orchestra.run.smithi162.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-06T14:17:19.720 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-06T14:17:19.720 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-09-06T14:17:19.918 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-4704.g8b38c 12/12 2024-09-06T14:17:19.919 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-4704.g8b38c 1/12 2024-09-06T14:17:19.919 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-06T14:17:19.919 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-06T14:17:19.919 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-06T14:17:19.919 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-06T14:17:19.919 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-06T14:17:19.919 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-06T14:17:19.919 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-06T14:17:19.919 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-06T14:17:19.919 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-06T14:17:19.919 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-06T14:17:19.970 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-fuse 2024-09-06T14:17:20.359 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-06T14:17:20.359 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:17:20.359 INFO:teuthology.orchestra.run.smithi142.stdout:Installed: 2024-09-06T14:17:20.359 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr-diskprediction-local-2:19.3.0-4704.g8b38c33c.el9.noarch 2024-09-06T14:17:20.360 INFO:teuthology.orchestra.run.smithi142.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-06T14:17:20.360 INFO:teuthology.orchestra.run.smithi142.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-06T14:17:20.360 INFO:teuthology.orchestra.run.smithi142.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-06T14:17:20.360 INFO:teuthology.orchestra.run.smithi142.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-06T14:17:20.360 INFO:teuthology.orchestra.run.smithi142.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-06T14:17:20.360 INFO:teuthology.orchestra.run.smithi142.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-06T14:17:20.360 INFO:teuthology.orchestra.run.smithi142.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-06T14:17:20.360 INFO:teuthology.orchestra.run.smithi142.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-06T14:17:20.360 INFO:teuthology.orchestra.run.smithi142.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-06T14:17:20.360 INFO:teuthology.orchestra.run.smithi142.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-06T14:17:20.361 INFO:teuthology.orchestra.run.smithi142.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-06T14:17:20.361 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:17:20.361 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2024-09-06T14:17:20.523 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:28 ago on Fri 06 Sep 2024 02:15:52 PM UTC. 2024-09-06T14:17:20.656 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-09-06T14:17:20.657 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-06T14:17:20.657 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repository Size 2024-09-06T14:17:20.657 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-06T14:17:20.657 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-09-06T14:17:20.657 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-fuse x86_64 2:19.3.0-4704.g8b38c33c.el9 ceph 900 k 2024-09-06T14:17:20.658 INFO:teuthology.orchestra.run.smithi162.stdout:Installing dependencies: 2024-09-06T14:17:20.658 INFO:teuthology.orchestra.run.smithi162.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-06T14:17:20.658 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-06T14:17:20.658 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-09-06T14:17:20.658 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-06T14:17:20.658 INFO:teuthology.orchestra.run.smithi162.stdout:Install 2 Packages 2024-09-06T14:17:20.658 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-06T14:17:20.658 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 980 k 2024-09-06T14:17:20.658 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 2.9 M 2024-09-06T14:17:20.658 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-09-06T14:17:20.717 DEBUG:teuthology.orchestra.run.smithi142:> sudo yum -y install ceph-mgr-rook 2024-09-06T14:17:20.950 INFO:teuthology.orchestra.run.smithi162.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 430 kB/s | 80 kB 00:00 2024-09-06T14:17:21.050 INFO:teuthology.orchestra.run.smithi162.stdout:(2/2): ceph-fuse-19.3.0-4704.g8b38c33c.el9.x86_ 3.1 MB/s | 900 kB 00:00 2024-09-06T14:17:21.050 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-09-06T14:17:21.050 INFO:teuthology.orchestra.run.smithi162.stdout:Total 2.4 MB/s | 980 kB 00:00 2024-09-06T14:17:21.062 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-09-06T14:17:21.084 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-09-06T14:17:21.084 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-09-06T14:17:21.163 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-09-06T14:17:21.164 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-09-06T14:17:21.291 INFO:teuthology.orchestra.run.smithi142.stdout:Last metadata expiration check: 0:01:25 ago on Fri 06 Sep 2024 02:15:56 PM UTC. 2024-09-06T14:17:21.357 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-09-06T14:17:21.426 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2024-09-06T14:17:21.427 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-09-06T14:17:21.428 INFO:teuthology.orchestra.run.smithi142.stdout: Package Arch Version Repository Size 2024-09-06T14:17:21.428 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-09-06T14:17:21.428 INFO:teuthology.orchestra.run.smithi142.stdout:Installing: 2024-09-06T14:17:21.428 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr-rook noarch 2:19.3.0-4704.g8b38c33c.el9 ceph-noarch 49 k 2024-09-06T14:17:21.428 INFO:teuthology.orchestra.run.smithi142.stdout:Installing dependencies: 2024-09-06T14:17:21.428 INFO:teuthology.orchestra.run.smithi142.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-06T14:17:21.428 INFO:teuthology.orchestra.run.smithi142.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-06T14:17:21.428 INFO:teuthology.orchestra.run.smithi142.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-06T14:17:21.428 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-06T14:17:21.428 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-06T14:17:21.428 INFO:teuthology.orchestra.run.smithi142.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-06T14:17:21.428 INFO:teuthology.orchestra.run.smithi142.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-06T14:17:21.428 INFO:teuthology.orchestra.run.smithi142.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-06T14:17:21.429 INFO:teuthology.orchestra.run.smithi142.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-06T14:17:21.429 INFO:teuthology.orchestra.run.smithi142.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-06T14:17:21.429 INFO:teuthology.orchestra.run.smithi142.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-06T14:17:21.429 INFO:teuthology.orchestra.run.smithi142.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-06T14:17:21.429 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:17:21.429 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction Summary 2024-09-06T14:17:21.429 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-09-06T14:17:21.429 INFO:teuthology.orchestra.run.smithi142.stdout:Install 13 Packages 2024-09-06T14:17:21.429 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:17:21.430 INFO:teuthology.orchestra.run.smithi142.stdout:Total download size: 2.2 M 2024-09-06T14:17:21.430 INFO:teuthology.orchestra.run.smithi142.stdout:Installed size: 26 M 2024-09-06T14:17:21.430 INFO:teuthology.orchestra.run.smithi142.stdout:Downloading Packages: 2024-09-06T14:17:21.501 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-06T14:17:21.563 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-fuse-2:19.3.0-4704.g8b38c33c.el9.x86_64 2/2 2024-09-06T14:17:21.939 INFO:teuthology.orchestra.run.smithi142.stdout:(1/13): ceph-mgr-rook-19.3.0-4704.g8b38c33c.el9 309 kB/s | 49 kB 00:00 2024-09-06T14:17:22.040 INFO:teuthology.orchestra.run.smithi142.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 74 kB/s | 19 kB 00:00 2024-09-06T14:17:22.065 INFO:teuthology.orchestra.run.smithi142.stdout:(3/13): python3-jsonpatch-1.21-16.el9.noarch.rp 90 kB/s | 26 kB 00:00 2024-09-06T14:17:22.237 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-fuse-2:19.3.0-4704.g8b38c33c.el9.x86_64 2/2 2024-09-06T14:17:22.237 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-fuse-2:19.3.0-4704.g8b38c33c.el9.x86_64 1/2 2024-09-06T14:17:22.382 INFO:teuthology.orchestra.run.smithi142.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 464 kB/s | 159 kB 00:00 2024-09-06T14:17:22.490 INFO:teuthology.orchestra.run.smithi142.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 656 kB/s | 279 kB 00:00 2024-09-06T14:17:22.493 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-06T14:17:22.493 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-06T14:17:22.493 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-09-06T14:17:22.494 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-fuse-2:19.3.0-4704.g8b38c33c.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-06T14:17:22.494 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-06T14:17:22.494 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-09-06T14:17:22.515 INFO:teuthology.orchestra.run.smithi142.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 401 kB/s | 54 kB 00:00 2024-09-06T14:17:22.557 INFO:teuthology.orchestra.run.smithi142.stdout:(7/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 359 kB/s | 222 kB 00:00 2024-09-06T14:17:22.583 INFO:teuthology.orchestra.run.smithi142.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 349 kB/s | 32 kB 00:00 2024-09-06T14:17:22.616 INFO:teuthology.orchestra.run.smithi142.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 141 kB/s | 14 kB 00:00 2024-09-06T14:17:22.650 INFO:teuthology.orchestra.run.smithi142.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 1.8 MB/s | 59 kB 00:00 2024-09-06T14:17:22.683 INFO:teuthology.orchestra.run.smithi142.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 2.6 MB/s | 90 kB 00:00 2024-09-06T14:17:22.709 INFO:teuthology.orchestra.run.smithi142.stdout:(12/13): python3-google-auth-2.34.0-1.el9.noarc 1.4 MB/s | 220 kB 00:00 2024-09-06T14:17:22.742 INFO:teuthology.orchestra.run.smithi142.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.7 MB/s | 1.0 MB 00:00 2024-09-06T14:17:22.745 INFO:teuthology.orchestra.run.smithi142.stdout:-------------------------------------------------------------------------------- 2024-09-06T14:17:22.745 INFO:teuthology.orchestra.run.smithi142.stdout:Total 1.7 MB/s | 2.2 MB 00:01 2024-09-06T14:17:22.795 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-volume 2024-09-06T14:17:22.811 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction check 2024-09-06T14:17:22.825 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction check succeeded. 2024-09-06T14:17:22.826 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction test 2024-09-06T14:17:22.913 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction test succeeded. 2024-09-06T14:17:22.913 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction 2024-09-06T14:17:23.369 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:31 ago on Fri 06 Sep 2024 02:15:52 PM UTC. 2024-09-06T14:17:23.429 INFO:teuthology.orchestra.run.smithi142.stdout: Preparing : 1/1 2024-09-06T14:17:23.505 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-09-06T14:17:23.506 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-06T14:17:23.506 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repository Size 2024-09-06T14:17:23.506 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-06T14:17:23.506 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-09-06T14:17:23.506 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-volume noarch 2:19.3.0-4704.g8b38c33c.el9 ceph-noarch 285 k 2024-09-06T14:17:23.506 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-06T14:17:23.506 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-09-06T14:17:23.506 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-06T14:17:23.506 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-09-06T14:17:23.507 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-06T14:17:23.507 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 285 k 2024-09-06T14:17:23.507 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 1.3 M 2024-09-06T14:17:23.507 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-09-06T14:17:23.598 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-06T14:17:23.661 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-06T14:17:23.716 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-06T14:17:23.722 INFO:teuthology.orchestra.run.smithi162.stdout:ceph-volume-19.3.0-4704.g8b38c33c.el9.noarch.rp 1.3 MB/s | 285 kB 00:00 2024-09-06T14:17:23.724 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-09-06T14:17:23.724 INFO:teuthology.orchestra.run.smithi162.stdout:Total 1.3 MB/s | 285 kB 00:00 2024-09-06T14:17:23.724 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-09-06T14:17:23.732 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-09-06T14:17:23.732 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-09-06T14:17:23.755 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-09-06T14:17:23.756 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-09-06T14:17:23.761 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-06T14:17:23.819 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-06T14:17:23.930 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-06T14:17:23.983 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-09-06T14:17:24.032 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-06T14:17:24.050 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-volume-2:19.3.0-4704.g8b38c33c.el9.noarch 1/1 2024-09-06T14:17:24.066 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-volume-2:19.3.0-4704.g8b38c33c.el9.noarch 1/1 2024-09-06T14:17:24.066 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-06T14:17:24.066 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-06T14:17:24.066 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-06T14:17:24.089 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-06T14:17:24.821 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-06T14:17:24.868 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-06T14:17:24.901 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-06T14:17:24.950 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-06T14:17:24.988 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-mgr-rook-2:19.3.0-4704.g8b38c33c.el9.noarch 13/13 2024-09-06T14:17:25.101 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-volume-2:19.3.0-4704.g8b38c33c.el9.noarch 1/1 2024-09-06T14:17:25.101 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-06T14:17:25.101 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-09-06T14:17:25.101 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-volume-2:19.3.0-4704.g8b38c33c.el9.noarch 2024-09-06T14:17:25.101 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-06T14:17:25.101 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-09-06T14:17:25.386 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install librados-devel 2024-09-06T14:17:25.731 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-4704.g8b38c33c.el9.noarch 13/13 2024-09-06T14:17:25.732 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-mgr-rook-2:19.3.0-4704.g8b38c33c.el9.noarch 1/13 2024-09-06T14:17:25.732 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-06T14:17:25.732 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-06T14:17:25.732 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-06T14:17:25.732 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-06T14:17:25.732 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-06T14:17:25.732 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-06T14:17:25.733 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-06T14:17:25.733 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-06T14:17:25.733 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-06T14:17:25.733 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-06T14:17:25.733 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-06T14:17:25.959 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:33 ago on Fri 06 Sep 2024 02:15:52 PM UTC. 2024-09-06T14:17:26.055 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-06T14:17:26.055 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:17:26.055 INFO:teuthology.orchestra.run.smithi142.stdout:Installed: 2024-09-06T14:17:26.055 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr-rook-2:19.3.0-4704.g8b38c33c.el9.noarch 2024-09-06T14:17:26.055 INFO:teuthology.orchestra.run.smithi142.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-06T14:17:26.056 INFO:teuthology.orchestra.run.smithi142.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-06T14:17:26.056 INFO:teuthology.orchestra.run.smithi142.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-06T14:17:26.056 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-06T14:17:26.056 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-06T14:17:26.056 INFO:teuthology.orchestra.run.smithi142.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-06T14:17:26.056 INFO:teuthology.orchestra.run.smithi142.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-06T14:17:26.056 INFO:teuthology.orchestra.run.smithi142.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-06T14:17:26.056 INFO:teuthology.orchestra.run.smithi142.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-06T14:17:26.056 INFO:teuthology.orchestra.run.smithi142.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-06T14:17:26.056 INFO:teuthology.orchestra.run.smithi142.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-06T14:17:26.056 INFO:teuthology.orchestra.run.smithi142.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-06T14:17:26.056 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:17:26.056 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2024-09-06T14:17:26.094 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-09-06T14:17:26.094 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-06T14:17:26.094 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repo Size 2024-09-06T14:17:26.094 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-06T14:17:26.095 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-09-06T14:17:26.095 INFO:teuthology.orchestra.run.smithi162.stdout: librados-devel x86_64 2:19.3.0-4704.g8b38c33c.el9 ceph 127 k 2024-09-06T14:17:26.095 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-06T14:17:26.095 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-09-06T14:17:26.095 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-06T14:17:26.095 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-09-06T14:17:26.095 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-06T14:17:26.096 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 127 k 2024-09-06T14:17:26.096 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 456 k 2024-09-06T14:17:26.096 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-09-06T14:17:26.280 INFO:teuthology.orchestra.run.smithi162.stdout:librados-devel-19.3.0-4704.g8b38c33c.el9.x86_64 690 kB/s | 127 kB 00:00 2024-09-06T14:17:26.280 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-09-06T14:17:26.280 INFO:teuthology.orchestra.run.smithi162.stdout:Total 684 kB/s | 127 kB 00:00 2024-09-06T14:17:26.281 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-09-06T14:17:26.286 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-09-06T14:17:26.286 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-09-06T14:17:26.289 DEBUG:teuthology.orchestra.run.smithi142:> sudo yum -y install ceph-mgr-cephadm 2024-09-06T14:17:26.351 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-09-06T14:17:26.351 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-09-06T14:17:26.501 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-09-06T14:17:26.609 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : librados-devel-2:19.3.0-4704.g8b38c33c.el9.x86_64 1/1 2024-09-06T14:17:26.864 INFO:teuthology.orchestra.run.smithi142.stdout:Last metadata expiration check: 0:01:30 ago on Fri 06 Sep 2024 02:15:56 PM UTC. 2024-09-06T14:17:26.998 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2024-09-06T14:17:26.998 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-09-06T14:17:26.998 INFO:teuthology.orchestra.run.smithi142.stdout: Package Arch Version Repository Size 2024-09-06T14:17:26.999 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-09-06T14:17:26.999 INFO:teuthology.orchestra.run.smithi142.stdout:Installing: 2024-09-06T14:17:26.999 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr-cephadm noarch 2:19.3.0-4704.g8b38c33c.el9 ceph-noarch 154 k 2024-09-06T14:17:26.999 INFO:teuthology.orchestra.run.smithi142.stdout:Installing dependencies: 2024-09-06T14:17:26.999 INFO:teuthology.orchestra.run.smithi142.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-06T14:17:26.999 INFO:teuthology.orchestra.run.smithi142.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-06T14:17:26.999 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-06T14:17:26.999 INFO:teuthology.orchestra.run.smithi142.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-06T14:17:26.999 INFO:teuthology.orchestra.run.smithi142.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-06T14:17:26.999 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:17:26.999 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction Summary 2024-09-06T14:17:26.999 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-09-06T14:17:27.000 INFO:teuthology.orchestra.run.smithi142.stdout:Install 6 Packages 2024-09-06T14:17:27.000 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:17:27.000 INFO:teuthology.orchestra.run.smithi142.stdout:Total download size: 7.0 M 2024-09-06T14:17:27.000 INFO:teuthology.orchestra.run.smithi142.stdout:Installed size: 33 M 2024-09-06T14:17:27.000 INFO:teuthology.orchestra.run.smithi142.stdout:Downloading Packages: 2024-09-06T14:17:27.043 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: librados-devel-2:19.3.0-4704.g8b38c33c.el9.x86_64 1/1 2024-09-06T14:17:27.235 INFO:teuthology.orchestra.run.smithi142.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.5 MB/s | 248 kB 00:00 2024-09-06T14:17:27.302 INFO:teuthology.orchestra.run.smithi142.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 8.0 MB/s | 548 kB 00:00 2024-09-06T14:17:27.319 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librados-devel-2:19.3.0-4704.g8b38c33c.el9.x86_64 1/1 2024-09-06T14:17:27.319 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-06T14:17:27.319 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-09-06T14:17:27.319 INFO:teuthology.orchestra.run.smithi162.stdout: librados-devel-2:19.3.0-4704.g8b38c33c.el9.x86_64 2024-09-06T14:17:27.320 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-06T14:17:27.320 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-09-06T14:17:27.336 INFO:teuthology.orchestra.run.smithi142.stdout:(3/6): ceph-mgr-cephadm-19.3.0-4704.g8b38c33c.e 595 kB/s | 154 kB 00:00 2024-09-06T14:17:27.361 INFO:teuthology.orchestra.run.smithi142.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 987 kB/s | 58 kB 00:00 2024-09-06T14:17:27.403 INFO:teuthology.orchestra.run.smithi142.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 1.1 MB/s | 74 kB 00:00 2024-09-06T14:17:27.478 INFO:teuthology.orchestra.run.smithi142.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 15 MB/s | 6.0 MB 00:00 2024-09-06T14:17:27.481 INFO:teuthology.orchestra.run.smithi142.stdout:-------------------------------------------------------------------------------- 2024-09-06T14:17:27.481 INFO:teuthology.orchestra.run.smithi142.stdout:Total 15 MB/s | 7.0 MB 00:00 2024-09-06T14:17:27.552 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction check 2024-09-06T14:17:27.563 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction check succeeded. 2024-09-06T14:17:27.564 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction test 2024-09-06T14:17:27.628 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install libcephfs2 2024-09-06T14:17:27.664 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction test succeeded. 2024-09-06T14:17:27.665 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction 2024-09-06T14:17:27.978 INFO:teuthology.orchestra.run.smithi142.stdout: Preparing : 1/1 2024-09-06T14:17:28.165 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-06T14:17:28.198 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:36 ago on Fri 06 Sep 2024 02:15:52 PM UTC. 2024-09-06T14:17:28.222 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-06T14:17:28.287 INFO:teuthology.orchestra.run.smithi162.stdout:Package libcephfs2-2:19.3.0-4704.g8b38c33c.el9.x86_64 is already installed. 2024-09-06T14:17:28.330 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-09-06T14:17:28.331 INFO:teuthology.orchestra.run.smithi162.stdout:Nothing to do. 2024-09-06T14:17:28.331 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-09-06T14:17:28.430 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install libcephfs-devel 2024-09-06T14:17:28.795 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-06T14:17:28.884 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-06T14:17:28.952 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-06T14:17:28.984 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-4704.g8b38c33c.el9.noarch 6/6 2024-09-06T14:17:28.994 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:37 ago on Fri 06 Sep 2024 02:15:52 PM UTC. 2024-09-06T14:17:29.126 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-09-06T14:17:29.126 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-06T14:17:29.126 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repo Size 2024-09-06T14:17:29.127 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-06T14:17:29.127 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-09-06T14:17:29.127 INFO:teuthology.orchestra.run.smithi162.stdout: libcephfs-devel x86_64 2:19.3.0-4704.g8b38c33c.el9 ceph 32 k 2024-09-06T14:17:29.127 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-06T14:17:29.127 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-09-06T14:17:29.127 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-06T14:17:29.127 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-09-06T14:17:29.127 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-06T14:17:29.127 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 32 k 2024-09-06T14:17:29.127 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 142 k 2024-09-06T14:17:29.127 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-09-06T14:17:29.284 INFO:teuthology.orchestra.run.smithi162.stdout:libcephfs-devel-19.3.0-4704.g8b38c33c.el9.x86_6 202 kB/s | 32 kB 00:00 2024-09-06T14:17:29.285 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-09-06T14:17:29.285 INFO:teuthology.orchestra.run.smithi162.stdout:Total 201 kB/s | 32 kB 00:00 2024-09-06T14:17:29.285 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-09-06T14:17:29.289 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-09-06T14:17:29.289 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-09-06T14:17:29.293 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-09-06T14:17:29.293 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-09-06T14:17:29.379 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-09-06T14:17:29.488 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libcephfs-devel-2:19.3.0-4704.g8b38c33c.el9.x86_64 1/1 2024-09-06T14:17:29.746 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-4704.g8b38c33c.el9.noarch 6/6 2024-09-06T14:17:29.746 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-4704.g8b38c33c.el9.noarch 1/6 2024-09-06T14:17:29.746 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-06T14:17:29.746 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-06T14:17:29.747 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-06T14:17:29.747 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-06T14:17:29.822 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: libcephfs-devel-2:19.3.0-4704.g8b38c33c.el9.x86_64 1/1 2024-09-06T14:17:30.013 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-06T14:17:30.013 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:17:30.013 INFO:teuthology.orchestra.run.smithi142.stdout:Installed: 2024-09-06T14:17:30.014 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr-cephadm-2:19.3.0-4704.g8b38c33c.el9.noarch 2024-09-06T14:17:30.014 INFO:teuthology.orchestra.run.smithi142.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-06T14:17:30.014 INFO:teuthology.orchestra.run.smithi142.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-06T14:17:30.014 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-06T14:17:30.014 INFO:teuthology.orchestra.run.smithi142.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-06T14:17:30.014 INFO:teuthology.orchestra.run.smithi142.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-06T14:17:30.014 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:17:30.014 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2024-09-06T14:17:30.057 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libcephfs-devel-2:19.3.0-4704.g8b38c33c.el9.x86_64 1/1 2024-09-06T14:17:30.058 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-06T14:17:30.058 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-09-06T14:17:30.058 INFO:teuthology.orchestra.run.smithi162.stdout: libcephfs-devel-2:19.3.0-4704.g8b38c33c.el9.x86_64 2024-09-06T14:17:30.058 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-06T14:17:30.058 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-09-06T14:17:30.249 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install librados2 2024-09-06T14:17:30.257 DEBUG:teuthology.orchestra.run.smithi142:> sudo yum -y install ceph-fuse 2024-09-06T14:17:30.805 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:38 ago on Fri 06 Sep 2024 02:15:52 PM UTC. 2024-09-06T14:17:30.807 INFO:teuthology.orchestra.run.smithi142.stdout:Last metadata expiration check: 0:01:34 ago on Fri 06 Sep 2024 02:15:56 PM UTC. 2024-09-06T14:17:30.895 INFO:teuthology.orchestra.run.smithi162.stdout:Package librados2-2:19.3.0-4704.g8b38c33c.el9.x86_64 is already installed. 2024-09-06T14:17:30.941 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2024-09-06T14:17:30.941 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-09-06T14:17:30.942 INFO:teuthology.orchestra.run.smithi142.stdout: Package Arch Version Repository Size 2024-09-06T14:17:30.942 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-09-06T14:17:30.942 INFO:teuthology.orchestra.run.smithi142.stdout:Installing: 2024-09-06T14:17:30.942 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-fuse x86_64 2:19.3.0-4704.g8b38c33c.el9 ceph 900 k 2024-09-06T14:17:30.942 INFO:teuthology.orchestra.run.smithi142.stdout:Installing dependencies: 2024-09-06T14:17:30.942 INFO:teuthology.orchestra.run.smithi142.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-06T14:17:30.942 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:17:30.942 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction Summary 2024-09-06T14:17:30.942 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-09-06T14:17:30.942 INFO:teuthology.orchestra.run.smithi142.stdout:Install 2 Packages 2024-09-06T14:17:30.942 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:17:30.942 INFO:teuthology.orchestra.run.smithi142.stdout:Total download size: 980 k 2024-09-06T14:17:30.943 INFO:teuthology.orchestra.run.smithi142.stdout:Installed size: 2.9 M 2024-09-06T14:17:30.943 INFO:teuthology.orchestra.run.smithi142.stdout:Downloading Packages: 2024-09-06T14:17:30.943 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-09-06T14:17:30.943 INFO:teuthology.orchestra.run.smithi162.stdout:Nothing to do. 2024-09-06T14:17:30.944 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-09-06T14:17:31.012 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install librbd1 2024-09-06T14:17:31.081 INFO:teuthology.orchestra.run.smithi142.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 672 kB/s | 80 kB 00:00 2024-09-06T14:17:31.247 INFO:teuthology.orchestra.run.smithi142.stdout:(2/2): ceph-fuse-19.3.0-4704.g8b38c33c.el9.x86_ 3.1 MB/s | 900 kB 00:00 2024-09-06T14:17:31.248 INFO:teuthology.orchestra.run.smithi142.stdout:-------------------------------------------------------------------------------- 2024-09-06T14:17:31.248 INFO:teuthology.orchestra.run.smithi142.stdout:Total 3.1 MB/s | 980 kB 00:00 2024-09-06T14:17:31.259 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction check 2024-09-06T14:17:31.281 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction check succeeded. 2024-09-06T14:17:31.282 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction test 2024-09-06T14:17:31.361 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction test succeeded. 2024-09-06T14:17:31.361 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction 2024-09-06T14:17:31.550 INFO:teuthology.orchestra.run.smithi142.stdout: Preparing : 1/1 2024-09-06T14:17:31.565 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:39 ago on Fri 06 Sep 2024 02:15:52 PM UTC. 2024-09-06T14:17:31.655 INFO:teuthology.orchestra.run.smithi162.stdout:Package librbd1-2:19.3.0-4704.g8b38c33c.el9.x86_64 is already installed. 2024-09-06T14:17:31.691 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-06T14:17:31.698 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-09-06T14:17:31.699 INFO:teuthology.orchestra.run.smithi162.stdout:Nothing to do. 2024-09-06T14:17:31.699 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-09-06T14:17:31.753 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-fuse-2:19.3.0-4704.g8b38c33c.el9.x86_64 2/2 2024-09-06T14:17:31.770 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install python3-rados 2024-09-06T14:17:32.329 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:40 ago on Fri 06 Sep 2024 02:15:52 PM UTC. 2024-09-06T14:17:32.418 INFO:teuthology.orchestra.run.smithi162.stdout:Package python3-rados-2:19.3.0-4704.g8b38c33c.el9.x86_64 is already installed. 2024-09-06T14:17:32.452 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-fuse-2:19.3.0-4704.g8b38c33c.el9.x86_64 2/2 2024-09-06T14:17:32.452 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-fuse-2:19.3.0-4704.g8b38c33c.el9.x86_64 1/2 2024-09-06T14:17:32.461 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-09-06T14:17:32.462 INFO:teuthology.orchestra.run.smithi162.stdout:Nothing to do. 2024-09-06T14:17:32.462 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-09-06T14:17:32.535 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install python3-rgw 2024-09-06T14:17:32.732 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-06T14:17:32.732 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:17:32.732 INFO:teuthology.orchestra.run.smithi142.stdout:Installed: 2024-09-06T14:17:32.732 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-fuse-2:19.3.0-4704.g8b38c33c.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-06T14:17:32.733 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:17:32.733 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2024-09-06T14:17:33.009 DEBUG:teuthology.orchestra.run.smithi142:> sudo yum -y install ceph-volume 2024-09-06T14:17:33.091 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:41 ago on Fri 06 Sep 2024 02:15:52 PM UTC. 2024-09-06T14:17:33.181 INFO:teuthology.orchestra.run.smithi162.stdout:Package python3-rgw-2:19.3.0-4704.g8b38c33c.el9.x86_64 is already installed. 2024-09-06T14:17:33.224 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-09-06T14:17:33.225 INFO:teuthology.orchestra.run.smithi162.stdout:Nothing to do. 2024-09-06T14:17:33.225 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-09-06T14:17:33.295 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install python3-cephfs 2024-09-06T14:17:33.584 INFO:teuthology.orchestra.run.smithi142.stdout:Last metadata expiration check: 0:01:37 ago on Fri 06 Sep 2024 02:15:56 PM UTC. 2024-09-06T14:17:33.722 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2024-09-06T14:17:33.723 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-09-06T14:17:33.723 INFO:teuthology.orchestra.run.smithi142.stdout: Package Arch Version Repository Size 2024-09-06T14:17:33.723 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-09-06T14:17:33.723 INFO:teuthology.orchestra.run.smithi142.stdout:Installing: 2024-09-06T14:17:33.723 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-volume noarch 2:19.3.0-4704.g8b38c33c.el9 ceph-noarch 285 k 2024-09-06T14:17:33.723 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:17:33.723 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction Summary 2024-09-06T14:17:33.723 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-09-06T14:17:33.724 INFO:teuthology.orchestra.run.smithi142.stdout:Install 1 Package 2024-09-06T14:17:33.724 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:17:33.724 INFO:teuthology.orchestra.run.smithi142.stdout:Total download size: 285 k 2024-09-06T14:17:33.724 INFO:teuthology.orchestra.run.smithi142.stdout:Installed size: 1.3 M 2024-09-06T14:17:33.724 INFO:teuthology.orchestra.run.smithi142.stdout:Downloading Packages: 2024-09-06T14:17:33.858 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:41 ago on Fri 06 Sep 2024 02:15:52 PM UTC. 2024-09-06T14:17:33.947 INFO:teuthology.orchestra.run.smithi162.stdout:Package python3-cephfs-2:19.3.0-4704.g8b38c33c.el9.x86_64 is already installed. 2024-09-06T14:17:33.960 INFO:teuthology.orchestra.run.smithi142.stdout:ceph-volume-19.3.0-4704.g8b38c33c.el9.noarch.rp 1.2 MB/s | 285 kB 00:00 2024-09-06T14:17:33.961 INFO:teuthology.orchestra.run.smithi142.stdout:-------------------------------------------------------------------------------- 2024-09-06T14:17:33.961 INFO:teuthology.orchestra.run.smithi142.stdout:Total 1.2 MB/s | 285 kB 00:00 2024-09-06T14:17:33.961 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction check 2024-09-06T14:17:33.969 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction check succeeded. 2024-09-06T14:17:33.969 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction test 2024-09-06T14:17:33.991 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-09-06T14:17:33.991 INFO:teuthology.orchestra.run.smithi162.stdout:Nothing to do. 2024-09-06T14:17:33.991 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-09-06T14:17:33.992 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction test succeeded. 2024-09-06T14:17:33.992 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction 2024-09-06T14:17:34.059 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install python3-rbd 2024-09-06T14:17:34.213 INFO:teuthology.orchestra.run.smithi142.stdout: Preparing : 1/1 2024-09-06T14:17:34.287 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-volume-2:19.3.0-4704.g8b38c33c.el9.noarch 1/1 2024-09-06T14:17:34.303 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-volume-2:19.3.0-4704.g8b38c33c.el9.noarch 1/1 2024-09-06T14:17:34.303 INFO:teuthology.orchestra.run.smithi142.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-06T14:17:34.304 INFO:teuthology.orchestra.run.smithi142.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-06T14:17:34.304 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:17:34.620 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:42 ago on Fri 06 Sep 2024 02:15:52 PM UTC. 2024-09-06T14:17:34.709 INFO:teuthology.orchestra.run.smithi162.stdout:Package python3-rbd-2:19.3.0-4704.g8b38c33c.el9.x86_64 is already installed. 2024-09-06T14:17:34.752 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-09-06T14:17:34.753 INFO:teuthology.orchestra.run.smithi162.stdout:Nothing to do. 2024-09-06T14:17:34.753 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-09-06T14:17:34.825 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install rbd-fuse 2024-09-06T14:17:35.320 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-volume-2:19.3.0-4704.g8b38c33c.el9.noarch 1/1 2024-09-06T14:17:35.320 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:17:35.320 INFO:teuthology.orchestra.run.smithi142.stdout:Installed: 2024-09-06T14:17:35.320 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-volume-2:19.3.0-4704.g8b38c33c.el9.noarch 2024-09-06T14:17:35.320 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:17:35.320 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2024-09-06T14:17:35.385 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:43 ago on Fri 06 Sep 2024 02:15:52 PM UTC. 2024-09-06T14:17:35.518 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-09-06T14:17:35.518 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-06T14:17:35.518 INFO:teuthology.orchestra.run.smithi162.stdout: Package Architecture Version Repository Size 2024-09-06T14:17:35.519 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-06T14:17:35.519 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-09-06T14:17:35.519 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-fuse x86_64 2:19.3.0-4704.g8b38c33c.el9 ceph 86 k 2024-09-06T14:17:35.519 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-06T14:17:35.519 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-09-06T14:17:35.519 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-06T14:17:35.519 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-09-06T14:17:35.519 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-06T14:17:35.519 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 86 k 2024-09-06T14:17:35.519 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 231 k 2024-09-06T14:17:35.519 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-09-06T14:17:35.633 DEBUG:teuthology.orchestra.run.smithi142:> sudo yum -y install librados-devel 2024-09-06T14:17:35.721 INFO:teuthology.orchestra.run.smithi162.stdout:rbd-fuse-19.3.0-4704.g8b38c33c.el9.x86_64.rpm 425 kB/s | 86 kB 00:00 2024-09-06T14:17:35.721 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-09-06T14:17:35.722 INFO:teuthology.orchestra.run.smithi162.stdout:Total 422 kB/s | 86 kB 00:00 2024-09-06T14:17:35.722 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-09-06T14:17:35.729 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-09-06T14:17:35.729 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-09-06T14:17:35.793 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-09-06T14:17:35.793 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-09-06T14:17:35.932 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-09-06T14:17:36.033 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : rbd-fuse-2:19.3.0-4704.g8b38c33c.el9.x86_64 1/1 2024-09-06T14:17:36.209 INFO:teuthology.orchestra.run.smithi142.stdout:Last metadata expiration check: 0:01:40 ago on Fri 06 Sep 2024 02:15:56 PM UTC. 2024-09-06T14:17:36.349 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2024-09-06T14:17:36.350 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-09-06T14:17:36.350 INFO:teuthology.orchestra.run.smithi142.stdout: Package Arch Version Repo Size 2024-09-06T14:17:36.350 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-09-06T14:17:36.350 INFO:teuthology.orchestra.run.smithi142.stdout:Installing: 2024-09-06T14:17:36.350 INFO:teuthology.orchestra.run.smithi142.stdout: librados-devel x86_64 2:19.3.0-4704.g8b38c33c.el9 ceph 127 k 2024-09-06T14:17:36.350 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:17:36.350 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction Summary 2024-09-06T14:17:36.350 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-09-06T14:17:36.350 INFO:teuthology.orchestra.run.smithi142.stdout:Install 1 Package 2024-09-06T14:17:36.350 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:17:36.351 INFO:teuthology.orchestra.run.smithi142.stdout:Total download size: 127 k 2024-09-06T14:17:36.351 INFO:teuthology.orchestra.run.smithi142.stdout:Installed size: 456 k 2024-09-06T14:17:36.351 INFO:teuthology.orchestra.run.smithi142.stdout:Downloading Packages: 2024-09-06T14:17:36.427 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: rbd-fuse-2:19.3.0-4704.g8b38c33c.el9.x86_64 1/1 2024-09-06T14:17:36.557 INFO:teuthology.orchestra.run.smithi142.stdout:librados-devel-19.3.0-4704.g8b38c33c.el9.x86_64 615 kB/s | 127 kB 00:00 2024-09-06T14:17:36.558 INFO:teuthology.orchestra.run.smithi142.stdout:-------------------------------------------------------------------------------- 2024-09-06T14:17:36.558 INFO:teuthology.orchestra.run.smithi142.stdout:Total 609 kB/s | 127 kB 00:00 2024-09-06T14:17:36.558 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction check 2024-09-06T14:17:36.564 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction check succeeded. 2024-09-06T14:17:36.564 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction test 2024-09-06T14:17:36.629 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction test succeeded. 2024-09-06T14:17:36.630 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction 2024-09-06T14:17:36.723 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : rbd-fuse-2:19.3.0-4704.g8b38c33c.el9.x86_64 1/1 2024-09-06T14:17:36.724 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-06T14:17:36.724 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-09-06T14:17:36.724 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-fuse-2:19.3.0-4704.g8b38c33c.el9.x86_64 2024-09-06T14:17:36.724 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-06T14:17:36.724 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-09-06T14:17:36.812 INFO:teuthology.orchestra.run.smithi142.stdout: Preparing : 1/1 2024-09-06T14:17:36.912 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : librados-devel-2:19.3.0-4704.g8b38c33c.el9.x86_64 1/1 2024-09-06T14:17:36.990 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install rbd-mirror 2024-09-06T14:17:37.369 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: librados-devel-2:19.3.0-4704.g8b38c33c.el9.x86_64 1/1 2024-09-06T14:17:37.575 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:45 ago on Fri 06 Sep 2024 02:15:52 PM UTC. 2024-09-06T14:17:37.621 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : librados-devel-2:19.3.0-4704.g8b38c33c.el9.x86_64 1/1 2024-09-06T14:17:37.622 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:17:37.622 INFO:teuthology.orchestra.run.smithi142.stdout:Installed: 2024-09-06T14:17:37.622 INFO:teuthology.orchestra.run.smithi142.stdout: librados-devel-2:19.3.0-4704.g8b38c33c.el9.x86_64 2024-09-06T14:17:37.622 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:17:37.622 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2024-09-06T14:17:37.716 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-09-06T14:17:37.717 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-06T14:17:37.717 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repo Size 2024-09-06T14:17:37.717 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-06T14:17:37.717 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-09-06T14:17:37.717 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-mirror x86_64 2:19.3.0-4704.g8b38c33c.el9 ceph 3.2 M 2024-09-06T14:17:37.717 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-06T14:17:37.718 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-09-06T14:17:37.718 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-06T14:17:37.718 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-09-06T14:17:37.718 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-06T14:17:37.718 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 3.2 M 2024-09-06T14:17:37.718 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 14 M 2024-09-06T14:17:37.718 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-09-06T14:17:37.869 DEBUG:teuthology.orchestra.run.smithi142:> sudo yum -y install libcephfs2 2024-09-06T14:17:38.043 INFO:teuthology.orchestra.run.smithi162.stdout:rbd-mirror-19.3.0-4704.g8b38c33c.el9.x86_64.rpm 9.8 MB/s | 3.2 MB 00:00 2024-09-06T14:17:38.044 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-09-06T14:17:38.044 INFO:teuthology.orchestra.run.smithi162.stdout:Total 9.7 MB/s | 3.2 MB 00:00 2024-09-06T14:17:38.044 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-09-06T14:17:38.054 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-09-06T14:17:38.054 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-09-06T14:17:38.139 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-09-06T14:17:38.140 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-09-06T14:17:38.436 INFO:teuthology.orchestra.run.smithi142.stdout:Last metadata expiration check: 0:01:42 ago on Fri 06 Sep 2024 02:15:56 PM UTC. 2024-09-06T14:17:38.526 INFO:teuthology.orchestra.run.smithi142.stdout:Package libcephfs2-2:19.3.0-4704.g8b38c33c.el9.x86_64 is already installed. 2024-09-06T14:17:38.572 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2024-09-06T14:17:38.573 INFO:teuthology.orchestra.run.smithi142.stdout:Nothing to do. 2024-09-06T14:17:38.573 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2024-09-06T14:17:38.603 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-09-06T14:17:38.640 DEBUG:teuthology.orchestra.run.smithi142:> sudo yum -y install libcephfs-devel 2024-09-06T14:17:38.677 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : rbd-mirror-2:19.3.0-4704.g8b38c33c.el9.x86_64 1/1 2024-09-06T14:17:38.703 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: rbd-mirror-2:19.3.0-4704.g8b38c33c.el9.x86_64 1/1 2024-09-06T14:17:38.703 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-06T14:17:38.703 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-06T14:17:38.703 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-06T14:17:38.703 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-06T14:17:38.703 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-06T14:17:39.205 INFO:teuthology.orchestra.run.smithi142.stdout:Last metadata expiration check: 0:01:43 ago on Fri 06 Sep 2024 02:15:56 PM UTC. 2024-09-06T14:17:39.337 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2024-09-06T14:17:39.338 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-09-06T14:17:39.338 INFO:teuthology.orchestra.run.smithi142.stdout: Package Arch Version Repo Size 2024-09-06T14:17:39.338 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-09-06T14:17:39.338 INFO:teuthology.orchestra.run.smithi142.stdout:Installing: 2024-09-06T14:17:39.338 INFO:teuthology.orchestra.run.smithi142.stdout: libcephfs-devel x86_64 2:19.3.0-4704.g8b38c33c.el9 ceph 32 k 2024-09-06T14:17:39.338 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:17:39.338 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction Summary 2024-09-06T14:17:39.338 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-09-06T14:17:39.338 INFO:teuthology.orchestra.run.smithi142.stdout:Install 1 Package 2024-09-06T14:17:39.338 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:17:39.339 INFO:teuthology.orchestra.run.smithi142.stdout:Total download size: 32 k 2024-09-06T14:17:39.339 INFO:teuthology.orchestra.run.smithi142.stdout:Installed size: 142 k 2024-09-06T14:17:39.339 INFO:teuthology.orchestra.run.smithi142.stdout:Downloading Packages: 2024-09-06T14:17:39.486 INFO:teuthology.orchestra.run.smithi142.stdout:libcephfs-devel-19.3.0-4704.g8b38c33c.el9.x86_6 215 kB/s | 32 kB 00:00 2024-09-06T14:17:39.487 INFO:teuthology.orchestra.run.smithi142.stdout:-------------------------------------------------------------------------------- 2024-09-06T14:17:39.487 INFO:teuthology.orchestra.run.smithi142.stdout:Total 213 kB/s | 32 kB 00:00 2024-09-06T14:17:39.487 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction check 2024-09-06T14:17:39.491 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction check succeeded. 2024-09-06T14:17:39.491 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction test 2024-09-06T14:17:39.495 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction test succeeded. 2024-09-06T14:17:39.495 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction 2024-09-06T14:17:39.574 INFO:teuthology.orchestra.run.smithi142.stdout: Preparing : 1/1 2024-09-06T14:17:39.691 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : libcephfs-devel-2:19.3.0-4704.g8b38c33c.el9.x86_64 1/1 2024-09-06T14:17:39.798 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : rbd-mirror-2:19.3.0-4704.g8b38c33c.el9.x86_64 1/1 2024-09-06T14:17:39.798 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-06T14:17:39.798 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-09-06T14:17:39.799 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-mirror-2:19.3.0-4704.g8b38c33c.el9.x86_64 2024-09-06T14:17:39.799 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-06T14:17:39.799 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-09-06T14:17:40.031 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install rbd-nbd 2024-09-06T14:17:40.036 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: libcephfs-devel-2:19.3.0-4704.g8b38c33c.el9.x86_64 1/1 2024-09-06T14:17:40.252 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libcephfs-devel-2:19.3.0-4704.g8b38c33c.el9.x86_64 1/1 2024-09-06T14:17:40.252 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:17:40.252 INFO:teuthology.orchestra.run.smithi142.stdout:Installed: 2024-09-06T14:17:40.252 INFO:teuthology.orchestra.run.smithi142.stdout: libcephfs-devel-2:19.3.0-4704.g8b38c33c.el9.x86_64 2024-09-06T14:17:40.253 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:17:40.253 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2024-09-06T14:17:40.442 DEBUG:teuthology.orchestra.run.smithi142:> sudo yum -y install librados2 2024-09-06T14:17:40.598 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:48 ago on Fri 06 Sep 2024 02:15:52 PM UTC. 2024-09-06T14:17:40.735 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-09-06T14:17:40.736 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-06T14:17:40.736 INFO:teuthology.orchestra.run.smithi162.stdout: Package Architecture Version Repository Size 2024-09-06T14:17:40.736 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-06T14:17:40.736 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-09-06T14:17:40.736 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-nbd x86_64 2:19.3.0-4704.g8b38c33c.el9 ceph 170 k 2024-09-06T14:17:40.737 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-06T14:17:40.737 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-09-06T14:17:40.737 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-06T14:17:40.737 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-09-06T14:17:40.737 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-06T14:17:40.737 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 170 k 2024-09-06T14:17:40.737 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 490 k 2024-09-06T14:17:40.737 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-09-06T14:17:40.948 INFO:teuthology.orchestra.run.smithi162.stdout:rbd-nbd-19.3.0-4704.g8b38c33c.el9.x86_64.rpm 810 kB/s | 170 kB 00:00 2024-09-06T14:17:40.949 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-09-06T14:17:40.949 INFO:teuthology.orchestra.run.smithi162.stdout:Total 803 kB/s | 170 kB 00:00 2024-09-06T14:17:40.949 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-09-06T14:17:40.957 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-09-06T14:17:40.957 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-09-06T14:17:41.003 INFO:teuthology.orchestra.run.smithi142.stdout:Last metadata expiration check: 0:01:45 ago on Fri 06 Sep 2024 02:15:56 PM UTC. 2024-09-06T14:17:41.020 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-09-06T14:17:41.021 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-09-06T14:17:41.093 INFO:teuthology.orchestra.run.smithi142.stdout:Package librados2-2:19.3.0-4704.g8b38c33c.el9.x86_64 is already installed. 2024-09-06T14:17:41.137 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2024-09-06T14:17:41.138 INFO:teuthology.orchestra.run.smithi142.stdout:Nothing to do. 2024-09-06T14:17:41.138 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2024-09-06T14:17:41.174 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-09-06T14:17:41.219 DEBUG:teuthology.orchestra.run.smithi142:> sudo yum -y install librbd1 2024-09-06T14:17:41.286 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : rbd-nbd-2:19.3.0-4704.g8b38c33c.el9.x86_64 1/1 2024-09-06T14:17:41.710 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: rbd-nbd-2:19.3.0-4704.g8b38c33c.el9.x86_64 1/1 2024-09-06T14:17:41.785 INFO:teuthology.orchestra.run.smithi142.stdout:Last metadata expiration check: 0:01:45 ago on Fri 06 Sep 2024 02:15:56 PM UTC. 2024-09-06T14:17:41.874 INFO:teuthology.orchestra.run.smithi142.stdout:Package librbd1-2:19.3.0-4704.g8b38c33c.el9.x86_64 is already installed. 2024-09-06T14:17:41.917 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2024-09-06T14:17:41.918 INFO:teuthology.orchestra.run.smithi142.stdout:Nothing to do. 2024-09-06T14:17:41.919 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2024-09-06T14:17:41.986 DEBUG:teuthology.orchestra.run.smithi142:> sudo yum -y install python3-rados 2024-09-06T14:17:41.990 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : rbd-nbd-2:19.3.0-4704.g8b38c33c.el9.x86_64 1/1 2024-09-06T14:17:41.990 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-06T14:17:41.990 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-09-06T14:17:41.990 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-nbd-2:19.3.0-4704.g8b38c33c.el9.x86_64 2024-09-06T14:17:41.990 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-06T14:17:41.990 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-09-06T14:17:42.220 DEBUG:teuthology.parallel:result is None 2024-09-06T14:17:42.541 INFO:teuthology.orchestra.run.smithi142.stdout:Last metadata expiration check: 0:01:46 ago on Fri 06 Sep 2024 02:15:56 PM UTC. 2024-09-06T14:17:42.629 INFO:teuthology.orchestra.run.smithi142.stdout:Package python3-rados-2:19.3.0-4704.g8b38c33c.el9.x86_64 is already installed. 2024-09-06T14:17:42.673 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2024-09-06T14:17:42.674 INFO:teuthology.orchestra.run.smithi142.stdout:Nothing to do. 2024-09-06T14:17:42.674 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2024-09-06T14:17:42.754 DEBUG:teuthology.orchestra.run.smithi142:> sudo yum -y install python3-rgw 2024-09-06T14:17:43.315 INFO:teuthology.orchestra.run.smithi142.stdout:Last metadata expiration check: 0:01:47 ago on Fri 06 Sep 2024 02:15:56 PM UTC. 2024-09-06T14:17:43.404 INFO:teuthology.orchestra.run.smithi142.stdout:Package python3-rgw-2:19.3.0-4704.g8b38c33c.el9.x86_64 is already installed. 2024-09-06T14:17:43.447 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2024-09-06T14:17:43.448 INFO:teuthology.orchestra.run.smithi142.stdout:Nothing to do. 2024-09-06T14:17:43.448 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2024-09-06T14:17:43.520 DEBUG:teuthology.orchestra.run.smithi142:> sudo yum -y install python3-cephfs 2024-09-06T14:17:44.085 INFO:teuthology.orchestra.run.smithi142.stdout:Last metadata expiration check: 0:01:48 ago on Fri 06 Sep 2024 02:15:56 PM UTC. 2024-09-06T14:17:44.176 INFO:teuthology.orchestra.run.smithi142.stdout:Package python3-cephfs-2:19.3.0-4704.g8b38c33c.el9.x86_64 is already installed. 2024-09-06T14:17:44.219 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2024-09-06T14:17:44.220 INFO:teuthology.orchestra.run.smithi142.stdout:Nothing to do. 2024-09-06T14:17:44.220 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2024-09-06T14:17:44.286 DEBUG:teuthology.orchestra.run.smithi142:> sudo yum -y install python3-rbd 2024-09-06T14:17:44.843 INFO:teuthology.orchestra.run.smithi142.stdout:Last metadata expiration check: 0:01:48 ago on Fri 06 Sep 2024 02:15:56 PM UTC. 2024-09-06T14:17:44.932 INFO:teuthology.orchestra.run.smithi142.stdout:Package python3-rbd-2:19.3.0-4704.g8b38c33c.el9.x86_64 is already installed. 2024-09-06T14:17:44.975 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2024-09-06T14:17:44.975 INFO:teuthology.orchestra.run.smithi142.stdout:Nothing to do. 2024-09-06T14:17:44.976 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2024-09-06T14:17:45.057 DEBUG:teuthology.orchestra.run.smithi142:> sudo yum -y install rbd-fuse 2024-09-06T14:17:45.611 INFO:teuthology.orchestra.run.smithi142.stdout:Last metadata expiration check: 0:01:49 ago on Fri 06 Sep 2024 02:15:56 PM UTC. 2024-09-06T14:17:45.745 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2024-09-06T14:17:45.746 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-09-06T14:17:45.746 INFO:teuthology.orchestra.run.smithi142.stdout: Package Architecture Version Repository Size 2024-09-06T14:17:45.746 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-09-06T14:17:45.746 INFO:teuthology.orchestra.run.smithi142.stdout:Installing: 2024-09-06T14:17:45.746 INFO:teuthology.orchestra.run.smithi142.stdout: rbd-fuse x86_64 2:19.3.0-4704.g8b38c33c.el9 ceph 86 k 2024-09-06T14:17:45.746 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:17:45.746 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction Summary 2024-09-06T14:17:45.747 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-09-06T14:17:45.747 INFO:teuthology.orchestra.run.smithi142.stdout:Install 1 Package 2024-09-06T14:17:45.747 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:17:45.747 INFO:teuthology.orchestra.run.smithi142.stdout:Total download size: 86 k 2024-09-06T14:17:45.747 INFO:teuthology.orchestra.run.smithi142.stdout:Installed size: 231 k 2024-09-06T14:17:45.747 INFO:teuthology.orchestra.run.smithi142.stdout:Downloading Packages: 2024-09-06T14:17:45.949 INFO:teuthology.orchestra.run.smithi142.stdout:rbd-fuse-19.3.0-4704.g8b38c33c.el9.x86_64.rpm 426 kB/s | 86 kB 00:00 2024-09-06T14:17:45.949 INFO:teuthology.orchestra.run.smithi142.stdout:-------------------------------------------------------------------------------- 2024-09-06T14:17:45.949 INFO:teuthology.orchestra.run.smithi142.stdout:Total 423 kB/s | 86 kB 00:00 2024-09-06T14:17:45.949 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction check 2024-09-06T14:17:45.955 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction check succeeded. 2024-09-06T14:17:45.956 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction test 2024-09-06T14:17:46.020 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction test succeeded. 2024-09-06T14:17:46.020 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction 2024-09-06T14:17:46.162 INFO:teuthology.orchestra.run.smithi142.stdout: Preparing : 1/1 2024-09-06T14:17:46.270 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : rbd-fuse-2:19.3.0-4704.g8b38c33c.el9.x86_64 1/1 2024-09-06T14:17:46.616 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: rbd-fuse-2:19.3.0-4704.g8b38c33c.el9.x86_64 1/1 2024-09-06T14:17:46.847 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : rbd-fuse-2:19.3.0-4704.g8b38c33c.el9.x86_64 1/1 2024-09-06T14:17:46.847 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:17:46.847 INFO:teuthology.orchestra.run.smithi142.stdout:Installed: 2024-09-06T14:17:46.847 INFO:teuthology.orchestra.run.smithi142.stdout: rbd-fuse-2:19.3.0-4704.g8b38c33c.el9.x86_64 2024-09-06T14:17:46.847 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:17:46.847 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2024-09-06T14:17:47.084 DEBUG:teuthology.orchestra.run.smithi142:> sudo yum -y install rbd-mirror 2024-09-06T14:17:47.663 INFO:teuthology.orchestra.run.smithi142.stdout:Last metadata expiration check: 0:01:51 ago on Fri 06 Sep 2024 02:15:56 PM UTC. 2024-09-06T14:17:47.804 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2024-09-06T14:17:47.805 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-09-06T14:17:47.805 INFO:teuthology.orchestra.run.smithi142.stdout: Package Arch Version Repo Size 2024-09-06T14:17:47.805 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-09-06T14:17:47.805 INFO:teuthology.orchestra.run.smithi142.stdout:Installing: 2024-09-06T14:17:47.805 INFO:teuthology.orchestra.run.smithi142.stdout: rbd-mirror x86_64 2:19.3.0-4704.g8b38c33c.el9 ceph 3.2 M 2024-09-06T14:17:47.805 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:17:47.805 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction Summary 2024-09-06T14:17:47.806 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-09-06T14:17:47.806 INFO:teuthology.orchestra.run.smithi142.stdout:Install 1 Package 2024-09-06T14:17:47.806 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:17:47.806 INFO:teuthology.orchestra.run.smithi142.stdout:Total download size: 3.2 M 2024-09-06T14:17:47.806 INFO:teuthology.orchestra.run.smithi142.stdout:Installed size: 14 M 2024-09-06T14:17:47.806 INFO:teuthology.orchestra.run.smithi142.stdout:Downloading Packages: 2024-09-06T14:17:48.135 INFO:teuthology.orchestra.run.smithi142.stdout:rbd-mirror-19.3.0-4704.g8b38c33c.el9.x86_64.rpm 9.6 MB/s | 3.2 MB 00:00 2024-09-06T14:17:48.137 INFO:teuthology.orchestra.run.smithi142.stdout:-------------------------------------------------------------------------------- 2024-09-06T14:17:48.137 INFO:teuthology.orchestra.run.smithi142.stdout:Total 9.6 MB/s | 3.2 MB 00:00 2024-09-06T14:17:48.137 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction check 2024-09-06T14:17:48.151 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction check succeeded. 2024-09-06T14:17:48.152 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction test 2024-09-06T14:17:48.251 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction test succeeded. 2024-09-06T14:17:48.251 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction 2024-09-06T14:17:48.714 INFO:teuthology.orchestra.run.smithi142.stdout: Preparing : 1/1 2024-09-06T14:17:48.770 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : rbd-mirror-2:19.3.0-4704.g8b38c33c.el9.x86_64 1/1 2024-09-06T14:17:48.795 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: rbd-mirror-2:19.3.0-4704.g8b38c33c.el9.x86_64 1/1 2024-09-06T14:17:48.795 INFO:teuthology.orchestra.run.smithi142.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-06T14:17:48.795 INFO:teuthology.orchestra.run.smithi142.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-06T14:17:48.795 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-06T14:17:48.795 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-06T14:17:48.796 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:17:49.893 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : rbd-mirror-2:19.3.0-4704.g8b38c33c.el9.x86_64 1/1 2024-09-06T14:17:49.893 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:17:49.893 INFO:teuthology.orchestra.run.smithi142.stdout:Installed: 2024-09-06T14:17:49.893 INFO:teuthology.orchestra.run.smithi142.stdout: rbd-mirror-2:19.3.0-4704.g8b38c33c.el9.x86_64 2024-09-06T14:17:49.893 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:17:49.893 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2024-09-06T14:17:50.140 DEBUG:teuthology.orchestra.run.smithi142:> sudo yum -y install rbd-nbd 2024-09-06T14:17:50.719 INFO:teuthology.orchestra.run.smithi142.stdout:Last metadata expiration check: 0:01:54 ago on Fri 06 Sep 2024 02:15:56 PM UTC. 2024-09-06T14:17:50.853 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2024-09-06T14:17:50.854 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-09-06T14:17:50.854 INFO:teuthology.orchestra.run.smithi142.stdout: Package Architecture Version Repository Size 2024-09-06T14:17:50.854 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-09-06T14:17:50.854 INFO:teuthology.orchestra.run.smithi142.stdout:Installing: 2024-09-06T14:17:50.854 INFO:teuthology.orchestra.run.smithi142.stdout: rbd-nbd x86_64 2:19.3.0-4704.g8b38c33c.el9 ceph 170 k 2024-09-06T14:17:50.854 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:17:50.854 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction Summary 2024-09-06T14:17:50.855 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-09-06T14:17:50.855 INFO:teuthology.orchestra.run.smithi142.stdout:Install 1 Package 2024-09-06T14:17:50.855 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:17:50.855 INFO:teuthology.orchestra.run.smithi142.stdout:Total download size: 170 k 2024-09-06T14:17:50.855 INFO:teuthology.orchestra.run.smithi142.stdout:Installed size: 490 k 2024-09-06T14:17:50.855 INFO:teuthology.orchestra.run.smithi142.stdout:Downloading Packages: 2024-09-06T14:17:51.081 INFO:teuthology.orchestra.run.smithi142.stdout:rbd-nbd-19.3.0-4704.g8b38c33c.el9.x86_64.rpm 755 kB/s | 170 kB 00:00 2024-09-06T14:17:51.082 INFO:teuthology.orchestra.run.smithi142.stdout:-------------------------------------------------------------------------------- 2024-09-06T14:17:51.082 INFO:teuthology.orchestra.run.smithi142.stdout:Total 749 kB/s | 170 kB 00:00 2024-09-06T14:17:51.083 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction check 2024-09-06T14:17:51.090 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction check succeeded. 2024-09-06T14:17:51.090 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction test 2024-09-06T14:17:51.154 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction test succeeded. 2024-09-06T14:17:51.155 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction 2024-09-06T14:17:51.303 INFO:teuthology.orchestra.run.smithi142.stdout: Preparing : 1/1 2024-09-06T14:17:51.402 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : rbd-nbd-2:19.3.0-4704.g8b38c33c.el9.x86_64 1/1 2024-09-06T14:17:51.812 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: rbd-nbd-2:19.3.0-4704.g8b38c33c.el9.x86_64 1/1 2024-09-06T14:17:52.071 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : rbd-nbd-2:19.3.0-4704.g8b38c33c.el9.x86_64 1/1 2024-09-06T14:17:52.071 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:17:52.071 INFO:teuthology.orchestra.run.smithi142.stdout:Installed: 2024-09-06T14:17:52.071 INFO:teuthology.orchestra.run.smithi142.stdout: rbd-nbd-2:19.3.0-4704.g8b38c33c.el9.x86_64 2024-09-06T14:17:52.071 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:17:52.071 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2024-09-06T14:17:52.320 DEBUG:teuthology.parallel:result is None 2024-09-06T14:17:52.321 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 2024-09-06T14:17:52.474 DEBUG:teuthology.orchestra.run.smithi142:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-06T14:17:52.496 INFO:teuthology.orchestra.run.smithi142.stdout:19.3.0-4704.g8b38c33c.el9 2024-09-06T14:17:52.497 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4704.g8b38c33c.el9 2024-09-06T14:17:52.497 INFO:teuthology.task.install:The correct ceph version 19.3.0-4704.g8b38c33c is installed. 2024-09-06T14:17:52.499 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 2024-09-06T14:17:52.645 DEBUG:teuthology.orchestra.run.smithi162:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-06T14:17:52.670 INFO:teuthology.orchestra.run.smithi162.stdout:19.3.0-4704.g8b38c33c.el9 2024-09-06T14:17:52.671 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4704.g8b38c33c.el9 2024-09-06T14:17:52.671 INFO:teuthology.task.install:The correct ceph version 19.3.0-4704.g8b38c33c is installed. 2024-09-06T14:17:52.673 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-06T14:17:52.673 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-09-06T14:17:52.673 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-06T14:17:52.699 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-09-06T14:17:52.699 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-06T14:17:52.741 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-06T14:17:52.742 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-09-06T14:17:52.742 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd of=/usr/bin/daemon-helper 2024-09-06T14:17:52.767 DEBUG:teuthology.orchestra.run.smithi142:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-06T14:17:52.831 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-09-06T14:17:52.831 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/usr/bin/daemon-helper 2024-09-06T14:17:52.861 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-06T14:17:52.929 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-06T14:17:52.929 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-09-06T14:17:52.929 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-06T14:17:52.955 DEBUG:teuthology.orchestra.run.smithi142:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-06T14:17:53.020 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-09-06T14:17:53.020 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-06T14:17:53.048 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-06T14:17:53.115 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-06T14:17:53.115 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-09-06T14:17:53.115 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd of=/usr/bin/stdin-killer 2024-09-06T14:17:53.140 DEBUG:teuthology.orchestra.run.smithi142:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-06T14:17:53.205 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-09-06T14:17:53.206 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/usr/bin/stdin-killer 2024-09-06T14:17:53.232 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-06T14:17:53.302 INFO:teuthology.run_tasks:Running task cephadm... 2024-09-06T14:17:53.402 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'client': {'debug ms': 1}, 'global': {'mon election default strategy': 1, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms type': 'async'}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon warn on pool no app': False}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd class default list': '*', 'osd class load list': '*', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'reached quota', 'but it is still running', 'overall HEALTH_', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', 'CEPHADM_STRAY_DAEMON'], 'log-only-match': ['CEPHADM_'], 'sha1': '8b38c33cecf4b0b7a9b3116eb9089e21e444ac73', 'cephadm_mode': 'cephadm-package'} 2024-09-06T14:17:53.402 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 2024-09-06T14:17:53.403 INFO:tasks.cephadm:Cluster fsid is ce32462e-6c5a-11ef-bce4-c7b262605968 2024-09-06T14:17:53.403 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-09-06T14:17:53.403 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.142', 'mon.c': '[v2:172.21.15.142:3301,v1:172.21.15.142:6790]', 'mon.b': '172.21.15.162'} 2024-09-06T14:17:53.403 INFO:tasks.cephadm:First mon is mon.a on smithi142 2024-09-06T14:17:53.403 INFO:tasks.cephadm:First mgr is y 2024-09-06T14:17:53.404 INFO:tasks.cephadm:Normalizing hostnames... 2024-09-06T14:17:53.404 DEBUG:teuthology.orchestra.run.smithi142:> sudo hostname $(hostname -s) 2024-09-06T14:17:53.429 DEBUG:teuthology.orchestra.run.smithi162:> sudo hostname $(hostname -s) 2024-09-06T14:17:53.467 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 on all hosts... 2024-09-06T14:17:53.467 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 pull 2024-09-06T14:17:53.472 DEBUG:teuthology.orchestra.run.smithi162:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 pull 2024-09-06T14:17:55.103 INFO:teuthology.orchestra.run.smithi142.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73... 2024-09-06T14:17:55.103 INFO:teuthology.orchestra.run.smithi162.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73... 2024-09-06T14:18:51.404 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-06T14:18:51.405 INFO:teuthology.orchestra.run.smithi162.stdout: "ceph_version": "ceph version 19.3.0-4704-g8b38c33c (8b38c33cecf4b0b7a9b3116eb9089e21e444ac73) squid (dev)", 2024-09-06T14:18:51.405 INFO:teuthology.orchestra.run.smithi162.stdout: "image_id": "1c45aca0610a312233aa2613f7b416b2884ae14779f1297041485ab28f1c8110", 2024-09-06T14:18:51.405 INFO:teuthology.orchestra.run.smithi162.stdout: "repo_digests": [ 2024-09-06T14:18:51.405 INFO:teuthology.orchestra.run.smithi162.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c86edd48571ab22373283621eb9feebce9df9c291d4164cfc4df1a09493e1f30" 2024-09-06T14:18:51.405 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-06T14:18:51.406 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-06T14:18:53.398 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-06T14:18:53.398 INFO:teuthology.orchestra.run.smithi142.stdout: "ceph_version": "ceph version 19.3.0-4704-g8b38c33c (8b38c33cecf4b0b7a9b3116eb9089e21e444ac73) squid (dev)", 2024-09-06T14:18:53.398 INFO:teuthology.orchestra.run.smithi142.stdout: "image_id": "1c45aca0610a312233aa2613f7b416b2884ae14779f1297041485ab28f1c8110", 2024-09-06T14:18:53.398 INFO:teuthology.orchestra.run.smithi142.stdout: "repo_digests": [ 2024-09-06T14:18:53.398 INFO:teuthology.orchestra.run.smithi142.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c86edd48571ab22373283621eb9feebce9df9c291d4164cfc4df1a09493e1f30" 2024-09-06T14:18:53.398 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-06T14:18:53.398 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-06T14:18:53.427 DEBUG:teuthology.orchestra.run.smithi142:> sudo mkdir -p /etc/ceph 2024-09-06T14:18:53.473 DEBUG:teuthology.orchestra.run.smithi162:> sudo mkdir -p /etc/ceph 2024-09-06T14:18:53.506 DEBUG:teuthology.orchestra.run.smithi142:> sudo chmod 777 /etc/ceph 2024-09-06T14:18:53.538 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod 777 /etc/ceph 2024-09-06T14:18:53.575 INFO:tasks.cephadm:Writing seed config... 2024-09-06T14:18:53.577 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-09-06T14:18:53.577 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-09-06T14:18:53.577 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-09-06T14:18:53.577 INFO:tasks.cephadm: override: [global] mon election default strategy = 1 2024-09-06T14:18:53.577 INFO:tasks.cephadm: override: [global] ms bind msgr1 = False 2024-09-06T14:18:53.577 INFO:tasks.cephadm: override: [global] ms bind msgr2 = True 2024-09-06T14:18:53.577 INFO:tasks.cephadm: override: [global] ms type = async 2024-09-06T14:18:53.577 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-09-06T14:18:53.578 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-09-06T14:18:53.578 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-09-06T14:18:53.578 INFO:tasks.cephadm: override: [mon] mon warn on pool no app = False 2024-09-06T14:18:53.578 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-09-06T14:18:53.578 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-09-06T14:18:53.578 INFO:tasks.cephadm: override: [osd] osd class default list = * 2024-09-06T14:18:53.578 INFO:tasks.cephadm: override: [osd] osd class load list = * 2024-09-06T14:18:53.578 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-09-06T14:18:53.579 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-09-06T14:18:53.579 DEBUG:teuthology.orchestra.run.smithi142:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-09-06T14:18:53.594 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = ce32462e-6c5a-11ef-bce4-c7b262605968 mon election default strategy = 1 ms bind msgr1 = False ms bind msgr2 = True ms type = async [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 osd class default list = * osd class load list = * [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon warn on pool no app = False [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] debug ms = 1 2024-09-06T14:18:53.596 DEBUG:teuthology.orchestra.run.smithi142:mon.a> sudo journalctl -f -n 0 -u ceph-ce32462e-6c5a-11ef-bce4-c7b262605968@mon.a.service 2024-09-06T14:18:53.637 DEBUG:teuthology.orchestra.run.smithi142:mgr.y> sudo journalctl -f -n 0 -u ceph-ce32462e-6c5a-11ef-bce4-c7b262605968@mgr.y.service 2024-09-06T14:18:53.680 INFO:tasks.cephadm:Bootstrapping... 2024-09-06T14:18:53.680 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 -v bootstrap --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.142 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-09-06T14:18:53.882 INFO:teuthology.orchestra.run.smithi142.stdout:-------------------------------------------------------------------------------- 2024-09-06T14:18:53.882 INFO:teuthology.orchestra.run.smithi142.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73', '-v', 'bootstrap', '--fsid', 'ce32462e-6c5a-11ef-bce4-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.142', '--skip-admin-label'] 2024-09-06T14:18:53.882 INFO:teuthology.orchestra.run.smithi142.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-09-06T14:18:53.882 INFO:teuthology.orchestra.run.smithi142.stdout:Verifying podman|docker is present... 2024-09-06T14:18:53.906 INFO:teuthology.orchestra.run.smithi142.stdout:/bin/podman: stdout 5.2.2 2024-09-06T14:18:53.906 INFO:teuthology.orchestra.run.smithi142.stdout:Verifying lvm2 is present... 2024-09-06T14:18:53.906 INFO:teuthology.orchestra.run.smithi142.stdout:Verifying time synchronization is in place... 2024-09-06T14:18:53.914 INFO:teuthology.orchestra.run.smithi142.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-06T14:18:53.914 INFO:teuthology.orchestra.run.smithi142.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-06T14:18:53.920 INFO:teuthology.orchestra.run.smithi142.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-06T14:18:53.920 INFO:teuthology.orchestra.run.smithi142.stdout:systemctl: stdout inactive 2024-09-06T14:18:53.926 INFO:teuthology.orchestra.run.smithi142.stdout:systemctl: stdout enabled 2024-09-06T14:18:53.933 INFO:teuthology.orchestra.run.smithi142.stdout:systemctl: stdout active 2024-09-06T14:18:53.933 INFO:teuthology.orchestra.run.smithi142.stdout:Unit chronyd.service is enabled and running 2024-09-06T14:18:53.933 INFO:teuthology.orchestra.run.smithi142.stdout:Repeating the final host check... 2024-09-06T14:18:53.961 INFO:teuthology.orchestra.run.smithi142.stdout:/bin/podman: stdout 5.2.2 2024-09-06T14:18:53.962 INFO:teuthology.orchestra.run.smithi142.stdout:podman (/bin/podman) version 5.2.2 is present 2024-09-06T14:18:53.962 INFO:teuthology.orchestra.run.smithi142.stdout:systemctl is present 2024-09-06T14:18:53.962 INFO:teuthology.orchestra.run.smithi142.stdout:lvcreate is present 2024-09-06T14:18:53.968 INFO:teuthology.orchestra.run.smithi142.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-06T14:18:53.968 INFO:teuthology.orchestra.run.smithi142.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-06T14:18:53.974 INFO:teuthology.orchestra.run.smithi142.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-06T14:18:53.974 INFO:teuthology.orchestra.run.smithi142.stdout:systemctl: stdout inactive 2024-09-06T14:18:53.981 INFO:teuthology.orchestra.run.smithi142.stdout:systemctl: stdout enabled 2024-09-06T14:18:53.988 INFO:teuthology.orchestra.run.smithi142.stdout:systemctl: stdout active 2024-09-06T14:18:53.988 INFO:teuthology.orchestra.run.smithi142.stdout:Unit chronyd.service is enabled and running 2024-09-06T14:18:53.988 INFO:teuthology.orchestra.run.smithi142.stdout:Host looks OK 2024-09-06T14:18:53.988 INFO:teuthology.orchestra.run.smithi142.stdout:Cluster fsid: ce32462e-6c5a-11ef-bce4-c7b262605968 2024-09-06T14:18:53.988 INFO:teuthology.orchestra.run.smithi142.stdout:Acquiring lock 140161033377968 on /run/cephadm/ce32462e-6c5a-11ef-bce4-c7b262605968.lock 2024-09-06T14:18:53.988 INFO:teuthology.orchestra.run.smithi142.stdout:Lock 140161033377968 acquired on /run/cephadm/ce32462e-6c5a-11ef-bce4-c7b262605968.lock 2024-09-06T14:18:53.988 INFO:teuthology.orchestra.run.smithi142.stdout:Verifying IP 172.21.15.142 port 3300 ... 2024-09-06T14:18:53.989 INFO:teuthology.orchestra.run.smithi142.stdout:Verifying IP 172.21.15.142 port 6789 ... 2024-09-06T14:18:53.989 INFO:teuthology.orchestra.run.smithi142.stdout:Base mon IP(s) is [172.21.15.142:3300, 172.21.15.142:6789], mon addrv is [v2:172.21.15.142:3300,v1:172.21.15.142:6789] 2024-09-06T14:18:53.992 INFO:teuthology.orchestra.run.smithi142.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.142 metric 100 2024-09-06T14:18:53.992 INFO:teuthology.orchestra.run.smithi142.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.142 metric 100 2024-09-06T14:18:53.994 INFO:teuthology.orchestra.run.smithi142.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-09-06T14:18:53.994 INFO:teuthology.orchestra.run.smithi142.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 1024 pref medium 2024-09-06T14:18:53.994 INFO:teuthology.orchestra.run.smithi142.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 100 pref medium 2024-09-06T14:18:53.997 INFO:teuthology.orchestra.run.smithi142.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-09-06T14:18:53.997 INFO:teuthology.orchestra.run.smithi142.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-09-06T14:18:53.998 INFO:teuthology.orchestra.run.smithi142.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-06T14:18:53.998 INFO:teuthology.orchestra.run.smithi142.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-09-06T14:18:53.998 INFO:teuthology.orchestra.run.smithi142.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe88:6e0d/64 scope link noprefixroute 2024-09-06T14:18:53.998 INFO:teuthology.orchestra.run.smithi142.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-06T14:18:53.998 INFO:teuthology.orchestra.run.smithi142.stdout:Mon IP `172.21.15.142` is in CIDR network `172.21.0.0/20` 2024-09-06T14:18:53.998 INFO:teuthology.orchestra.run.smithi142.stdout:Mon IP `172.21.15.142` is in CIDR network `172.21.0.0/20` 2024-09-06T14:18:53.998 INFO:teuthology.orchestra.run.smithi142.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-09-06T14:18:53.998 INFO:teuthology.orchestra.run.smithi142.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-09-06T14:18:53.999 INFO:teuthology.orchestra.run.smithi142.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73... 2024-09-06T14:18:54.492 INFO:teuthology.orchestra.run.smithi142.stdout:/bin/podman: stdout 1c45aca0610a312233aa2613f7b416b2884ae14779f1297041485ab28f1c8110 2024-09-06T14:18:54.493 INFO:teuthology.orchestra.run.smithi142.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73... 2024-09-06T14:18:54.493 INFO:teuthology.orchestra.run.smithi142.stdout:/bin/podman: stderr Getting image source signatures 2024-09-06T14:18:54.493 INFO:teuthology.orchestra.run.smithi142.stdout:/bin/podman: stderr Copying blob sha256:975930033597cc619449a5e495a30a9586fba9aa8e75bb850a8a1511772810f3 2024-09-06T14:18:54.493 INFO:teuthology.orchestra.run.smithi142.stdout:/bin/podman: stderr Copying blob sha256:e8b54c863393b7a8216a71737771b73b16a8e43ec7acf927c7faa175c255e551 2024-09-06T14:18:54.493 INFO:teuthology.orchestra.run.smithi142.stdout:/bin/podman: stderr Copying config sha256:1c45aca0610a312233aa2613f7b416b2884ae14779f1297041485ab28f1c8110 2024-09-06T14:18:54.493 INFO:teuthology.orchestra.run.smithi142.stdout:/bin/podman: stderr Writing manifest to image destination 2024-09-06T14:18:55.430 INFO:teuthology.orchestra.run.smithi142.stdout:ceph: stdout ceph version 19.3.0-4704-g8b38c33c (8b38c33cecf4b0b7a9b3116eb9089e21e444ac73) squid (dev) 2024-09-06T14:18:55.430 INFO:teuthology.orchestra.run.smithi142.stdout:Ceph version: ceph version 19.3.0-4704-g8b38c33c (8b38c33cecf4b0b7a9b3116eb9089e21e444ac73) squid (dev) 2024-09-06T14:18:55.430 INFO:teuthology.orchestra.run.smithi142.stdout:Extracting ceph user uid/gid from container image... 2024-09-06T14:18:56.294 INFO:teuthology.orchestra.run.smithi142.stdout:stat: stdout 167 167 2024-09-06T14:18:56.294 INFO:teuthology.orchestra.run.smithi142.stdout:Creating initial keys... 2024-09-06T14:18:57.333 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph-authtool: stdout AQBQD9tmBp05JRAAC+oYhnDv3gvBhWDvaXy9ZQ== 2024-09-06T14:18:58.346 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph-authtool: stdout AQBRD9tmrAwLJxAAh+L3V3GE6f/+iZeN1U4IjA== 2024-09-06T14:18:59.244 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph-authtool: stdout AQBSD9tmbD1mKBAAWN5X5t9gM+rZjFVC8UsOmw== 2024-09-06T14:18:59.244 INFO:teuthology.orchestra.run.smithi142.stdout:Creating initial monmap... 2024-09-06T14:19:08.241 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-06T14:19:08.242 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-09-06T14:19:08.242 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to ce32462e-6c5a-11ef-bce4-c7b262605968 2024-09-06T14:19:08.242 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-06T14:19:08.242 INFO:teuthology.orchestra.run.smithi142.stdout:monmaptool for a [v2:172.21.15.142:3300,v1:172.21.15.142:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-06T14:19:08.242 INFO:teuthology.orchestra.run.smithi142.stdout:setting min_mon_release = quincy 2024-09-06T14:19:08.242 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/monmaptool: set fsid to ce32462e-6c5a-11ef-bce4-c7b262605968 2024-09-06T14:19:08.242 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-06T14:19:08.242 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:19:08.242 INFO:teuthology.orchestra.run.smithi142.stdout:Creating mon... 2024-09-06T14:19:11.690 INFO:teuthology.orchestra.run.smithi142.stdout:create mon.a on 2024-09-06T14:19:11.893 INFO:teuthology.orchestra.run.smithi142.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-09-06T14:19:12.041 INFO:teuthology.orchestra.run.smithi142.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-09-06T14:19:12.230 INFO:teuthology.orchestra.run.smithi142.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-ce32462e-6c5a-11ef-bce4-c7b262605968.target → /etc/systemd/system/ceph-ce32462e-6c5a-11ef-bce4-c7b262605968.target. 2024-09-06T14:19:12.230 INFO:teuthology.orchestra.run.smithi142.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-ce32462e-6c5a-11ef-bce4-c7b262605968.target → /etc/systemd/system/ceph-ce32462e-6c5a-11ef-bce4-c7b262605968.target. 2024-09-06T14:19:12.470 INFO:teuthology.orchestra.run.smithi142.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-ce32462e-6c5a-11ef-bce4-c7b262605968@mon.a 2024-09-06T14:19:12.470 INFO:teuthology.orchestra.run.smithi142.stdout:systemctl: stderr Failed to reset failed state of unit ceph-ce32462e-6c5a-11ef-bce4-c7b262605968@mon.a.service: Unit ceph-ce32462e-6c5a-11ef-bce4-c7b262605968@mon.a.service not loaded. 2024-09-06T14:19:12.631 INFO:teuthology.orchestra.run.smithi142.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-ce32462e-6c5a-11ef-bce4-c7b262605968.target.wants/ceph-ce32462e-6c5a-11ef-bce4-c7b262605968@mon.a.service → /etc/systemd/system/ceph-ce32462e-6c5a-11ef-bce4-c7b262605968@.service. 2024-09-06T14:19:13.132 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:13 smithi142 ceph-mon[29589]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-06T14:19:13.146 INFO:teuthology.orchestra.run.smithi142.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-06T14:19:13.146 INFO:teuthology.orchestra.run.smithi142.stdout:systemctl: stdout disabled 2024-09-06T14:19:13.153 INFO:teuthology.orchestra.run.smithi142.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-06T14:19:13.154 INFO:teuthology.orchestra.run.smithi142.stdout:systemctl: stdout inactive 2024-09-06T14:19:13.154 INFO:teuthology.orchestra.run.smithi142.stdout:firewalld.service is not enabled 2024-09-06T14:19:13.154 INFO:teuthology.orchestra.run.smithi142.stdout:Not possible to enable service . firewalld.service is not available 2024-09-06T14:19:13.154 INFO:teuthology.orchestra.run.smithi142.stdout:Waiting for mon to start... 2024-09-06T14:19:13.154 INFO:teuthology.orchestra.run.smithi142.stdout:Waiting for mon... 2024-09-06T14:19:13.382 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:13 smithi142 bash[29570]: 74ffb78be2cf387d2c8061d02c903c24c6cdac486f1588acd5e651ac0c4fc30e 2024-09-06T14:19:13.383 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:13 smithi142 systemd[1]: Started Ceph mon.a for ce32462e-6c5a-11ef-bce4-c7b262605968. 2024-09-06T14:19:14.271 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout cluster: 2024-09-06T14:19:14.272 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout id: ce32462e-6c5a-11ef-bce4-c7b262605968 2024-09-06T14:19:14.272 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-09-06T14:19:14.272 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout 2024-09-06T14:19:14.272 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout services: 2024-09-06T14:19:14.272 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.502145s) 2024-09-06T14:19:14.272 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-09-06T14:19:14.272 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-09-06T14:19:14.272 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout 2024-09-06T14:19:14.272 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout data: 2024-09-06T14:19:14.272 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-09-06T14:19:14.272 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-09-06T14:19:14.272 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-09-06T14:19:14.273 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout pgs: 2024-09-06T14:19:14.273 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout 2024-09-06T14:19:14.273 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:13.534+0000 7fa1541e2640 1 Processor -- start 2024-09-06T14:19:14.273 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:13.534+0000 7fa1541e2640 1 -- start start 2024-09-06T14:19:14.273 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:13.534+0000 7fa1541e2640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa14c106710 0x7fa14c106b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:19:14.273 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:13.535+0000 7fa1541e2640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fa14c107100 con 0x7fa14c106710 2024-09-06T14:19:14.273 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:13.535+0000 7fa151f7e640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa14c106710 0x7fa14c106b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:19:14.273 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:13.535+0000 7fa151f7e640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa14c106710 0x7fa14c106b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:41424/0 (socket says 172.21.15.142:41424) 2024-09-06T14:19:14.273 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:13.535+0000 7fa151f7e640 1 -- 172.21.15.142:0/2441617887 learned_addr learned my addr 172.21.15.142:0/2441617887 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:19:14.273 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:13.535+0000 7fa151f7e640 1 -- 172.21.15.142:0/2441617887 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa14c107280 con 0x7fa14c106710 2024-09-06T14:19:14.273 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:13.537+0000 7fa151f7e640 1 --2- 172.21.15.142:0/2441617887 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa14c106710 0x7fa14c106b10 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fa148009820 tx=0x7fa14802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=196df8a72007f2a8 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:19:14.273 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:13.537+0000 7fa150f7c640 1 -- 172.21.15.142:0/2441617887 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa148037070 con 0x7fa14c106710 2024-09-06T14:19:14.273 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:13.537+0000 7fa150f7c640 1 -- 172.21.15.142:0/2441617887 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa148032440 con 0x7fa14c106710 2024-09-06T14:19:14.274 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:13.537+0000 7fa150f7c640 1 -- 172.21.15.142:0/2441617887 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa148032740 con 0x7fa14c106710 2024-09-06T14:19:14.274 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:13.538+0000 7fa1541e2640 1 -- 172.21.15.142:0/2441617887 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa14c106710 msgr2=0x7fa14c106b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:19:14.274 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:13.538+0000 7fa1541e2640 1 --2- 172.21.15.142:0/2441617887 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa14c106710 0x7fa14c106b10 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fa148009820 tx=0x7fa14802a270 comp rx=0 tx=0).stop 2024-09-06T14:19:14.274 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:13.538+0000 7fa1541e2640 1 -- 172.21.15.142:0/2441617887 shutdown_connections 2024-09-06T14:19:14.274 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:13.538+0000 7fa1541e2640 1 --2- 172.21.15.142:0/2441617887 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa14c106710 0x7fa14c106b10 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:19:14.274 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:13.538+0000 7fa1541e2640 1 -- 172.21.15.142:0/2441617887 >> 172.21.15.142:0/2441617887 conn(0x7fa14c102020 msgr2=0x7fa14c104440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:19:14.274 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:13.538+0000 7fa1541e2640 1 -- 172.21.15.142:0/2441617887 shutdown_connections 2024-09-06T14:19:14.274 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:13.538+0000 7fa1541e2640 1 -- 172.21.15.142:0/2441617887 wait complete. 2024-09-06T14:19:14.274 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:13.539+0000 7fa1541e2640 1 Processor -- start 2024-09-06T14:19:14.274 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:13.539+0000 7fa1541e2640 1 -- start start 2024-09-06T14:19:14.274 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:13.539+0000 7fa1541e2640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa14c106710 0x7fa14c199f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:19:14.274 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:13.539+0000 7fa1541e2640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fa14c107bd0 con 0x7fa14c106710 2024-09-06T14:19:14.274 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:13.539+0000 7fa151f7e640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa14c106710 0x7fa14c199f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:19:14.275 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:13.539+0000 7fa151f7e640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa14c106710 0x7fa14c199f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:41438/0 (socket says 172.21.15.142:41438) 2024-09-06T14:19:14.275 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:13.539+0000 7fa151f7e640 1 -- 172.21.15.142:0/1396206473 learned_addr learned my addr 172.21.15.142:0/1396206473 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:19:14.275 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:13.539+0000 7fa151f7e640 1 -- 172.21.15.142:0/1396206473 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa14c19a4d0 con 0x7fa14c106710 2024-09-06T14:19:14.275 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:13.539+0000 7fa151f7e640 1 --2- 172.21.15.142:0/1396206473 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa14c106710 0x7fa14c199f70 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fa1480090b0 tx=0x7fa1480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:19:14.275 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:13.540+0000 7fa142ffd640 1 -- 172.21.15.142:0/1396206473 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa148041070 con 0x7fa14c106710 2024-09-06T14:19:14.275 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:13.540+0000 7fa142ffd640 1 -- 172.21.15.142:0/1396206473 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa14803c7a0 con 0x7fa14c106710 2024-09-06T14:19:14.275 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:13.540+0000 7fa1541e2640 1 -- 172.21.15.142:0/1396206473 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa14c19a760 con 0x7fa14c106710 2024-09-06T14:19:14.275 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:13.540+0000 7fa142ffd640 1 -- 172.21.15.142:0/1396206473 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa148037040 con 0x7fa14c106710 2024-09-06T14:19:14.275 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:13.540+0000 7fa1541e2640 1 -- 172.21.15.142:0/1396206473 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa14c19b120 con 0x7fa14c106710 2024-09-06T14:19:14.275 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:13.540+0000 7fa142ffd640 1 -- 172.21.15.142:0/1396206473 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fa148044070 con 0x7fa14c106710 2024-09-06T14:19:14.275 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:13.540+0000 7fa142ffd640 1 -- 172.21.15.142:0/1396206473 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fa14803d3f0 con 0x7fa14c106710 2024-09-06T14:19:14.275 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:13.541+0000 7fa1541e2640 1 -- 172.21.15.142:0/1396206473 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa14c108380 con 0x7fa14c106710 2024-09-06T14:19:14.276 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:13.543+0000 7fa142ffd640 1 -- 172.21.15.142:0/1396206473 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7fa148046450 con 0x7fa14c106710 2024-09-06T14:19:14.276 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:13.603+0000 7fa1541e2640 1 -- 172.21.15.142:0/1396206473 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7fa14c19bec0 con 0x7fa14c106710 2024-09-06T14:19:14.276 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:13.603+0000 7fa142ffd640 1 -- 172.21.15.142:0/1396206473 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+317 (secure 0 0 0) 0x7fa148046630 con 0x7fa14c106710 2024-09-06T14:19:14.276 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:13.604+0000 7fa1541e2640 1 -- 172.21.15.142:0/1396206473 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa14c106710 msgr2=0x7fa14c199f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:19:14.276 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:13.604+0000 7fa1541e2640 1 --2- 172.21.15.142:0/1396206473 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa14c106710 0x7fa14c199f70 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fa1480090b0 tx=0x7fa1480046c0 comp rx=0 tx=0).stop 2024-09-06T14:19:14.276 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:13.604+0000 7fa1541e2640 1 -- 172.21.15.142:0/1396206473 shutdown_connections 2024-09-06T14:19:14.276 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:13.604+0000 7fa1541e2640 1 --2- 172.21.15.142:0/1396206473 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa14c106710 0x7fa14c199f70 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:19:14.276 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:13.604+0000 7fa1541e2640 1 -- 172.21.15.142:0/1396206473 >> 172.21.15.142:0/1396206473 conn(0x7fa14c102020 msgr2=0x7fa14c190a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:19:14.276 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:13.604+0000 7fa1541e2640 1 -- 172.21.15.142:0/1396206473 shutdown_connections 2024-09-06T14:19:14.276 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:13.604+0000 7fa1541e2640 1 -- 172.21.15.142:0/1396206473 wait complete. 2024-09-06T14:19:14.276 INFO:teuthology.orchestra.run.smithi142.stdout:mon is available 2024-09-06T14:19:14.276 INFO:teuthology.orchestra.run.smithi142.stdout:Assimilating anything we can from ceph.conf... 2024-09-06T14:19:14.381 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:14 smithi142 ceph-mon[29589]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-06T14:19:14.381 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:14 smithi142 ceph-mon[29589]: monmap epoch 1 2024-09-06T14:19:14.381 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:14 smithi142 ceph-mon[29589]: fsid ce32462e-6c5a-11ef-bce4-c7b262605968 2024-09-06T14:19:14.381 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:14 smithi142 ceph-mon[29589]: last_changed 2024-09-06T14:19:01.768092+0000 2024-09-06T14:19:14.381 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:14 smithi142 ceph-mon[29589]: created 2024-09-06T14:19:01.768092+0000 2024-09-06T14:19:14.381 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:14 smithi142 ceph-mon[29589]: min_mon_release 19 (squid) 2024-09-06T14:19:14.381 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:14 smithi142 ceph-mon[29589]: election_strategy: 1 2024-09-06T14:19:14.382 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:14 smithi142 ceph-mon[29589]: 0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-06T14:19:14.382 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:14 smithi142 ceph-mon[29589]: fsmap 2024-09-06T14:19:14.382 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:14 smithi142 ceph-mon[29589]: osdmap e1: 0 total, 0 up, 0 in 2024-09-06T14:19:14.382 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:14 smithi142 ceph-mon[29589]: mgrmap e1: no daemons active 2024-09-06T14:19:14.382 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:14 smithi142 ceph-mon[29589]: from='client.? 172.21.15.142:0/1396206473' entity='client.admin' cmd={"prefix": "status"} : dispatch 2024-09-06T14:19:15.328 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:15 smithi142 ceph-mon[29589]: from='client.? 172.21.15.142:0/3959149026' entity='client.admin' cmd={"prefix": "config assimilate-conf"} : dispatch 2024-09-06T14:19:15.329 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:15 smithi142 ceph-mon[29589]: from='client.? 172.21.15.142:0/3959149026' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-09-06T14:19:15.339 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout 2024-09-06T14:19:15.340 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout [global] 2024-09-06T14:19:15.340 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout fsid = ce32462e-6c5a-11ef-bce4-c7b262605968 2024-09-06T14:19:15.340 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-09-06T14:19:15.340 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.142:3300,v1:172.21.15.142:6789] 2024-09-06T14:19:15.340 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-06T14:19:15.340 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-06T14:19:15.341 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-06T14:19:15.341 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-06T14:19:15.341 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout 2024-09-06T14:19:15.341 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-06T14:19:15.341 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-06T14:19:15.341 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout 2024-09-06T14:19:15.342 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout [osd] 2024-09-06T14:19:15.342 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-06T14:19:15.342 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-06T14:19:15.342 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:14.673+0000 7f9846fb4640 1 Processor -- start 2024-09-06T14:19:15.342 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:14.673+0000 7f9846fb4640 1 -- start start 2024-09-06T14:19:15.342 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:14.674+0000 7f9846fb4640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f98401055b0 0x7f98401059b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:19:15.343 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:14.674+0000 7f9846fb4640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f9840105fa0 con 0x7f98401055b0 2024-09-06T14:19:15.343 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:14.674+0000 7f9844d50640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f98401055b0 0x7f98401059b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:19:15.343 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:14.674+0000 7f9844d50640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f98401055b0 0x7f98401059b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:41446/0 (socket says 172.21.15.142:41446) 2024-09-06T14:19:15.344 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:14.674+0000 7f9844d50640 1 -- 172.21.15.142:0/2282867957 learned_addr learned my addr 172.21.15.142:0/2282867957 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:19:15.344 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:14.674+0000 7f9844d50640 1 -- 172.21.15.142:0/2282867957 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9840106120 con 0x7f98401055b0 2024-09-06T14:19:15.344 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:14.675+0000 7f9844d50640 1 --2- 172.21.15.142:0/2282867957 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f98401055b0 0x7f98401059b0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f9834009820 tx=0x7f983402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=979a33783aebf162 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:19:15.344 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:14.675+0000 7f983f7fe640 1 -- 172.21.15.142:0/2282867957 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9834037070 con 0x7f98401055b0 2024-09-06T14:19:15.344 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:14.675+0000 7f983f7fe640 1 -- 172.21.15.142:0/2282867957 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9834032440 con 0x7f98401055b0 2024-09-06T14:19:15.344 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:14.675+0000 7f983f7fe640 1 -- 172.21.15.142:0/2282867957 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9834032740 con 0x7f98401055b0 2024-09-06T14:19:15.345 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:14.675+0000 7f9846fb4640 1 -- 172.21.15.142:0/2282867957 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f98401055b0 msgr2=0x7f98401059b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:19:15.345 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:14.675+0000 7f9846fb4640 1 --2- 172.21.15.142:0/2282867957 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f98401055b0 0x7f98401059b0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f9834009820 tx=0x7f983402a270 comp rx=0 tx=0).stop 2024-09-06T14:19:15.345 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:14.675+0000 7f9846fb4640 1 -- 172.21.15.142:0/2282867957 shutdown_connections 2024-09-06T14:19:15.345 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:14.675+0000 7f9846fb4640 1 --2- 172.21.15.142:0/2282867957 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f98401055b0 0x7f98401059b0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:19:15.345 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:14.675+0000 7f9846fb4640 1 -- 172.21.15.142:0/2282867957 >> 172.21.15.142:0/2282867957 conn(0x7f9840100f40 msgr2=0x7f9840103360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:19:15.346 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:14.676+0000 7f9846fb4640 1 -- 172.21.15.142:0/2282867957 shutdown_connections 2024-09-06T14:19:15.346 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:14.676+0000 7f9846fb4640 1 -- 172.21.15.142:0/2282867957 wait complete. 2024-09-06T14:19:15.346 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:14.676+0000 7f9846fb4640 1 Processor -- start 2024-09-06T14:19:15.346 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:14.676+0000 7f9846fb4640 1 -- start start 2024-09-06T14:19:15.347 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:14.676+0000 7f9846fb4640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f98401055b0 0x7f9840079870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:19:15.347 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:14.676+0000 7f9846fb4640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f9840106ac0 con 0x7f98401055b0 2024-09-06T14:19:15.347 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:14.676+0000 7f9844d50640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f98401055b0 0x7f9840079870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:19:15.347 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:14.676+0000 7f9844d50640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f98401055b0 0x7f9840079870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:41448/0 (socket says 172.21.15.142:41448) 2024-09-06T14:19:15.347 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:14.676+0000 7f9844d50640 1 -- 172.21.15.142:0/3959149026 learned_addr learned my addr 172.21.15.142:0/3959149026 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:19:15.348 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:14.677+0000 7f9844d50640 1 -- 172.21.15.142:0/3959149026 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9840079dd0 con 0x7f98401055b0 2024-09-06T14:19:15.348 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:14.677+0000 7f9844d50640 1 --2- 172.21.15.142:0/3959149026 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f98401055b0 0x7f9840079870 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f98340090b0 tx=0x7f98340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:19:15.348 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:14.677+0000 7f983dffb640 1 -- 172.21.15.142:0/3959149026 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9834041070 con 0x7f98401055b0 2024-09-06T14:19:15.348 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:14.677+0000 7f983dffb640 1 -- 172.21.15.142:0/3959149026 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f983403c7a0 con 0x7f98401055b0 2024-09-06T14:19:15.348 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:14.677+0000 7f983dffb640 1 -- 172.21.15.142:0/3959149026 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9834037040 con 0x7f98401055b0 2024-09-06T14:19:15.348 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:14.677+0000 7f9846fb4640 1 -- 172.21.15.142:0/3959149026 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f984019cc60 con 0x7f98401055b0 2024-09-06T14:19:15.349 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:14.677+0000 7f9846fb4640 1 -- 172.21.15.142:0/3959149026 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f984019d140 con 0x7f98401055b0 2024-09-06T14:19:15.349 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:14.677+0000 7f983dffb640 1 -- 172.21.15.142:0/3959149026 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f9834044070 con 0x7f98401055b0 2024-09-06T14:19:15.349 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:14.677+0000 7f983dffb640 1 -- 172.21.15.142:0/3959149026 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f983403d3f0 con 0x7f98401055b0 2024-09-06T14:19:15.349 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:14.678+0000 7f9846fb4640 1 -- 172.21.15.142:0/3959149026 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9840107270 con 0x7f98401055b0 2024-09-06T14:19:15.349 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:14.680+0000 7f983dffb640 1 -- 172.21.15.142:0/3959149026 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f9834046450 con 0x7f98401055b0 2024-09-06T14:19:15.350 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:14.740+0000 7f9846fb4640 1 -- 172.21.15.142:0/3959149026 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f984019d3c0 con 0x7f98401055b0 2024-09-06T14:19:15.350 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:14.743+0000 7f983dffb640 1 -- 172.21.15.142:0/3959149026 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+376 (secure 0 0 0) 0x7f9834046630 con 0x7f98401055b0 2024-09-06T14:19:15.350 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:14.744+0000 7f983dffb640 1 -- 172.21.15.142:0/3959149026 <== mon.0 v2:172.21.15.142:3300/0 8 ==== config(25 keys) ==== 1023+0+0 (secure 0 0 0) 0x7f9834058480 con 0x7f98401055b0 2024-09-06T14:19:15.350 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:14.744+0000 7f9846fb4640 1 -- 172.21.15.142:0/3959149026 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f98401055b0 msgr2=0x7f9840079870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:19:15.350 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:14.744+0000 7f9846fb4640 1 --2- 172.21.15.142:0/3959149026 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f98401055b0 0x7f9840079870 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f98340090b0 tx=0x7f98340046c0 comp rx=0 tx=0).stop 2024-09-06T14:19:15.350 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:14.744+0000 7f9846fb4640 1 -- 172.21.15.142:0/3959149026 shutdown_connections 2024-09-06T14:19:15.351 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:14.744+0000 7f9846fb4640 1 --2- 172.21.15.142:0/3959149026 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f98401055b0 0x7f9840079870 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:19:15.351 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:14.744+0000 7f9846fb4640 1 -- 172.21.15.142:0/3959149026 >> 172.21.15.142:0/3959149026 conn(0x7f9840100f40 msgr2=0x7f984018d8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:19:15.351 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:14.744+0000 7f9846fb4640 1 -- 172.21.15.142:0/3959149026 shutdown_connections 2024-09-06T14:19:15.351 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:14.744+0000 7f9846fb4640 1 -- 172.21.15.142:0/3959149026 wait complete. 2024-09-06T14:19:15.351 INFO:teuthology.orchestra.run.smithi142.stdout:Generating new minimal ceph.conf... 2024-09-06T14:19:16.456 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:16 smithi142 ceph-mon[29589]: from='client.? 172.21.15.142:0/2899179732' entity='client.admin' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:19:16.625 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:15.729+0000 7f5740eba640 1 Processor -- start 2024-09-06T14:19:16.625 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:15.730+0000 7f5740eba640 1 -- start start 2024-09-06T14:19:16.625 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:15.730+0000 7f5740eba640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f573c106710 0x7f573c106b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:19:16.625 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:15.730+0000 7f5740eba640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f573c107100 con 0x7f573c106710 2024-09-06T14:19:16.625 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:15.730+0000 7f573a59c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f573c106710 0x7f573c106b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:19:16.625 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:15.730+0000 7f573a59c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f573c106710 0x7f573c106b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:41452/0 (socket says 172.21.15.142:41452) 2024-09-06T14:19:16.625 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:15.730+0000 7f573a59c640 1 -- 172.21.15.142:0/57407830 learned_addr learned my addr 172.21.15.142:0/57407830 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:19:16.625 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:15.730+0000 7f573a59c640 1 -- 172.21.15.142:0/57407830 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f573c107280 con 0x7f573c106710 2024-09-06T14:19:16.625 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:15.731+0000 7f573a59c640 1 --2- 172.21.15.142:0/57407830 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f573c106710 0x7f573c106b10 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f572c009820 tx=0x7f572c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f96c8dfda60dd768 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:19:16.625 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:15.731+0000 7f573959a640 1 -- 172.21.15.142:0/57407830 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f572c037070 con 0x7f573c106710 2024-09-06T14:19:16.626 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:15.731+0000 7f573959a640 1 -- 172.21.15.142:0/57407830 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1023+0+0 (secure 0 0 0) 0x7f572c0328a0 con 0x7f573c106710 2024-09-06T14:19:16.626 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:15.731+0000 7f573959a640 1 -- 172.21.15.142:0/57407830 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f572c030690 con 0x7f573c106710 2024-09-06T14:19:16.626 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:15.732+0000 7f5740eba640 1 -- 172.21.15.142:0/57407830 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f573c106710 msgr2=0x7f573c106b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:19:16.626 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:15.732+0000 7f5740eba640 1 --2- 172.21.15.142:0/57407830 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f573c106710 0x7f573c106b10 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f572c009820 tx=0x7f572c02a270 comp rx=0 tx=0).stop 2024-09-06T14:19:16.626 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:15.732+0000 7f5740eba640 1 -- 172.21.15.142:0/57407830 shutdown_connections 2024-09-06T14:19:16.626 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:15.732+0000 7f5740eba640 1 --2- 172.21.15.142:0/57407830 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f573c106710 0x7f573c106b10 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:19:16.626 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:15.732+0000 7f5740eba640 1 -- 172.21.15.142:0/57407830 >> 172.21.15.142:0/57407830 conn(0x7f573c102020 msgr2=0x7f573c104440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:19:16.626 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:15.732+0000 7f5740eba640 1 -- 172.21.15.142:0/57407830 shutdown_connections 2024-09-06T14:19:16.627 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:15.732+0000 7f5740eba640 1 -- 172.21.15.142:0/57407830 wait complete. 2024-09-06T14:19:16.627 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:15.733+0000 7f5740eba640 1 Processor -- start 2024-09-06T14:19:16.627 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:15.733+0000 7f5740eba640 1 -- start start 2024-09-06T14:19:16.627 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:15.733+0000 7f5740eba640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f573c106710 0x7f573c19a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:19:16.627 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:15.733+0000 7f5740eba640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f573c10ab40 con 0x7f573c106710 2024-09-06T14:19:16.627 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:15.733+0000 7f573a59c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f573c106710 0x7f573c19a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:19:16.627 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:15.733+0000 7f573a59c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f573c106710 0x7f573c19a6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:41458/0 (socket says 172.21.15.142:41458) 2024-09-06T14:19:16.627 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:15.733+0000 7f573a59c640 1 -- 172.21.15.142:0/2899179732 learned_addr learned my addr 172.21.15.142:0/2899179732 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:19:16.627 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:15.733+0000 7f573a59c640 1 -- 172.21.15.142:0/2899179732 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f573c19ac40 con 0x7f573c106710 2024-09-06T14:19:16.628 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:15.733+0000 7f573a59c640 1 --2- 172.21.15.142:0/2899179732 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f573c106710 0x7f573c19a6e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f572c006f10 tx=0x7f572c032670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:19:16.628 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:15.734+0000 7f572b7fe640 1 -- 172.21.15.142:0/2899179732 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f572c037040 con 0x7f573c106710 2024-09-06T14:19:16.628 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:15.734+0000 7f572b7fe640 1 -- 172.21.15.142:0/2899179732 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1023+0+0 (secure 0 0 0) 0x7f572c040070 con 0x7f573c106710 2024-09-06T14:19:16.628 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:15.734+0000 7f5740eba640 1 -- 172.21.15.142:0/2899179732 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f573c19aed0 con 0x7f573c106710 2024-09-06T14:19:16.628 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:15.734+0000 7f572b7fe640 1 -- 172.21.15.142:0/2899179732 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f572c03a390 con 0x7f573c106710 2024-09-06T14:19:16.628 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:15.734+0000 7f5740eba640 1 -- 172.21.15.142:0/2899179732 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f573c19b890 con 0x7f573c106710 2024-09-06T14:19:16.628 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:15.734+0000 7f572b7fe640 1 -- 172.21.15.142:0/2899179732 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f572c03a650 con 0x7f573c106710 2024-09-06T14:19:16.628 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:15.734+0000 7f572b7fe640 1 -- 172.21.15.142:0/2899179732 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f572c0457c0 con 0x7f573c106710 2024-09-06T14:19:16.628 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:15.735+0000 7f5740eba640 1 -- 172.21.15.142:0/2899179732 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f573c106b90 con 0x7f573c106710 2024-09-06T14:19:16.628 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:15.737+0000 7f572b7fe640 1 -- 172.21.15.142:0/2899179732 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f572c032830 con 0x7f573c106710 2024-09-06T14:19:16.628 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:15.795+0000 7f5740eba640 1 -- 172.21.15.142:0/2899179732 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f573c19bea0 con 0x7f573c106710 2024-09-06T14:19:16.629 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:15.795+0000 7f572b7fe640 1 -- 172.21.15.142:0/2899179732 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+177 (secure 0 0 0) 0x7f572c03b430 con 0x7f573c106710 2024-09-06T14:19:16.629 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:15.796+0000 7f5740eba640 1 -- 172.21.15.142:0/2899179732 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f573c106710 msgr2=0x7f573c19a6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:19:16.629 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:15.796+0000 7f5740eba640 1 --2- 172.21.15.142:0/2899179732 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f573c106710 0x7f573c19a6e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f572c006f10 tx=0x7f572c032670 comp rx=0 tx=0).stop 2024-09-06T14:19:16.629 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:15.796+0000 7f5740eba640 1 -- 172.21.15.142:0/2899179732 shutdown_connections 2024-09-06T14:19:16.629 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:15.796+0000 7f5740eba640 1 --2- 172.21.15.142:0/2899179732 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f573c106710 0x7f573c19a6e0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:19:16.629 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:15.796+0000 7f5740eba640 1 -- 172.21.15.142:0/2899179732 >> 172.21.15.142:0/2899179732 conn(0x7f573c102020 msgr2=0x7f573c102b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:19:16.629 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:15.796+0000 7f5740eba640 1 -- 172.21.15.142:0/2899179732 shutdown_connections 2024-09-06T14:19:16.629 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:15.796+0000 7f5740eba640 1 -- 172.21.15.142:0/2899179732 wait complete. 2024-09-06T14:19:16.629 INFO:teuthology.orchestra.run.smithi142.stdout:Restarting the monitor... 2024-09-06T14:19:16.956 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:16 smithi142 systemd[1]: Stopping Ceph mon.a for ce32462e-6c5a-11ef-bce4-c7b262605968... 2024-09-06T14:19:17.706 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:17 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mon-a[29585]: 2024-09-06T14:19:17.424+0000 7fa7250de640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-06T14:19:17.706 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:17 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mon-a[29585]: 2024-09-06T14:19:17.424+0000 7fa7250de640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-09-06T14:19:17.707 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:17 smithi142 podman[29833]: 2024-09-06 14:19:17.568199557 +0000 UTC m=+0.767263438 container died 74ffb78be2cf387d2c8061d02c903c24c6cdac486f1588acd5e651ac0c4fc30e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mon-a, GIT_CLEAN=True, ceph=True, RELEASE=wip-athakkar-testing-2024-09-04-1837-8b38c33, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-06T14:19:18.193 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:17 smithi142 podman[29833]: 2024-09-06 14:19:17.93196862 +0000 UTC m=+1.131032492 container cleanup 74ffb78be2cf387d2c8061d02c903c24c6cdac486f1588acd5e651ac0c4fc30e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mon-a, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-athakkar-testing-2024-09-04-1837-8b38c33, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, io.buildah.version=1.37.2) 2024-09-06T14:19:18.193 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:17 smithi142 bash[29833]: ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mon-a 2024-09-06T14:19:18.445 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 podman[29846]: 2024-09-06 14:19:18.199577123 +0000 UTC m=+0.630820403 container remove 74ffb78be2cf387d2c8061d02c903c24c6cdac486f1588acd5e651ac0c4fc30e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mon-a, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-athakkar-testing-2024-09-04-1837-8b38c33, GIT_BRANCH=HEAD) 2024-09-06T14:19:18.445 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 systemd[1]: ceph-ce32462e-6c5a-11ef-bce4-c7b262605968@mon.a.service: Deactivated successfully. 2024-09-06T14:19:18.445 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 systemd[1]: Stopped Ceph mon.a for ce32462e-6c5a-11ef-bce4-c7b262605968. 2024-09-06T14:19:18.445 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 systemd[1]: Starting Ceph mon.a for ce32462e-6c5a-11ef-bce4-c7b262605968... 2024-09-06T14:19:18.706 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 podman[29939]: 2024-09-06 14:19:18.550309307 +0000 UTC m=+0.094172200 container create e23b8f60e296b972502999b12c5cbedbb4c39b4bb7bc67fe0b8ad0da8551e445 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mon-a, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-athakkar-testing-2024-09-04-1837-8b38c33, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731) 2024-09-06T14:19:18.707 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 podman[29939]: 2024-09-06 14:19:18.471962442 +0000 UTC m=+0.015825349 image pull 1c45aca0610a312233aa2613f7b416b2884ae14779f1297041485ab28f1c8110 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 2024-09-06T14:19:18.876 INFO:teuthology.orchestra.run.smithi142.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-09-06T14:19:19.081 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 podman[29939]: 2024-09-06 14:19:18.794409863 +0000 UTC m=+0.338272757 container init e23b8f60e296b972502999b12c5cbedbb4c39b4bb7bc67fe0b8ad0da8551e445 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mon-a, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_CLEAN=True, RELEASE=wip-athakkar-testing-2024-09-04-1837-8b38c33, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-09-06T14:19:19.081 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 podman[29939]: 2024-09-06 14:19:18.7975215 +0000 UTC m=+0.341384404 container start e23b8f60e296b972502999b12c5cbedbb4c39b4bb7bc67fe0b8ad0da8551e445 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mon-a, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-athakkar-testing-2024-09-04-1837-8b38c33, io.buildah.version=1.37.2) 2024-09-06T14:19:19.081 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: set uid:gid to 167:167 (ceph:ceph) 2024-09-06T14:19:19.081 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: ceph version 19.3.0-4704-g8b38c33c (8b38c33cecf4b0b7a9b3116eb9089e21e444ac73) squid (dev), process ceph-mon, pid 2 2024-09-06T14:19:19.082 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: pidfile_write: ignore empty --pid-file 2024-09-06T14:19:19.082 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: load: jerasure load: lrc 2024-09-06T14:19:19.082 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: RocksDB version: 7.9.2 2024-09-06T14:19:19.082 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Git sha 0 2024-09-06T14:19:19.082 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Compile date 2024-09-04 13:52:53 2024-09-06T14:19:19.082 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: DB SUMMARY 2024-09-06T14:19:19.082 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: DB Session ID: QPZ2N7JIH0C1PW4C03JI 2024-09-06T14:19:19.082 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: CURRENT file: CURRENT 2024-09-06T14:19:19.082 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: IDENTITY file: IDENTITY 2024-09-06T14:19:19.082 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-09-06T14:19:19.082 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-09-06T14:19:19.082 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 91920 ; 2024-09-06T14:19:19.082 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.error_if_exists: 0 2024-09-06T14:19:19.083 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.create_if_missing: 0 2024-09-06T14:19:19.083 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.paranoid_checks: 1 2024-09-06T14:19:19.083 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-06T14:19:19.083 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-06T14:19:19.083 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-06T14:19:19.083 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.env: 0x556f50512160 2024-09-06T14:19:19.083 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.fs: PosixFileSystem 2024-09-06T14:19:19.083 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.info_log: 0x556f51bdd880 2024-09-06T14:19:19.083 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.max_file_opening_threads: 16 2024-09-06T14:19:19.083 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.statistics: (nil) 2024-09-06T14:19:19.083 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.use_fsync: 0 2024-09-06T14:19:19.083 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.max_log_file_size: 0 2024-09-06T14:19:19.083 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-06T14:19:19.084 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-06T14:19:19.084 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.keep_log_file_num: 1000 2024-09-06T14:19:19.084 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.recycle_log_file_num: 0 2024-09-06T14:19:19.084 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.allow_fallocate: 1 2024-09-06T14:19:19.084 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.allow_mmap_reads: 0 2024-09-06T14:19:19.084 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.allow_mmap_writes: 0 2024-09-06T14:19:19.084 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.use_direct_reads: 0 2024-09-06T14:19:19.084 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-06T14:19:19.084 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.create_missing_column_families: 0 2024-09-06T14:19:19.084 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.db_log_dir: 2024-09-06T14:19:19.084 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.wal_dir: 2024-09-06T14:19:19.084 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-06T14:19:19.084 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-06T14:19:19.085 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-06T14:19:19.085 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-06T14:19:19.085 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-06T14:19:19.085 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-06T14:19:19.085 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.advise_random_on_open: 1 2024-09-06T14:19:19.085 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.db_write_buffer_size: 0 2024-09-06T14:19:19.085 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.write_buffer_manager: 0x556f51be0e60 2024-09-06T14:19:19.085 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-06T14:19:19.085 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-06T14:19:19.085 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-06T14:19:19.085 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.rate_limiter: (nil) 2024-09-06T14:19:19.085 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-06T14:19:19.085 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.wal_recovery_mode: 2 2024-09-06T14:19:19.086 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.enable_thread_tracking: 0 2024-09-06T14:19:19.086 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.enable_pipelined_write: 0 2024-09-06T14:19:19.086 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.unordered_write: 0 2024-09-06T14:19:19.086 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-06T14:19:19.086 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-06T14:19:19.086 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-06T14:19:19.086 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-06T14:19:19.086 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.row_cache: None 2024-09-06T14:19:19.086 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.wal_filter: None 2024-09-06T14:19:19.088 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-06T14:19:19.088 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.allow_ingest_behind: 0 2024-09-06T14:19:19.088 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.two_write_queues: 0 2024-09-06T14:19:19.088 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.manual_wal_flush: 0 2024-09-06T14:19:19.088 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.wal_compression: 0 2024-09-06T14:19:19.088 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.atomic_flush: 0 2024-09-06T14:19:19.088 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-06T14:19:19.088 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-06T14:19:19.089 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-06T14:19:19.089 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.log_readahead_size: 0 2024-09-06T14:19:19.089 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-06T14:19:19.089 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.best_efforts_recovery: 0 2024-09-06T14:19:19.089 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-06T14:19:19.089 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-06T14:19:19.089 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.allow_data_in_errors: 0 2024-09-06T14:19:19.089 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.db_host_id: __hostname__ 2024-09-06T14:19:19.089 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-06T14:19:19.089 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.max_background_jobs: 2 2024-09-06T14:19:19.089 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.max_background_compactions: -1 2024-09-06T14:19:19.090 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.max_subcompactions: 1 2024-09-06T14:19:19.090 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-06T14:19:19.090 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-06T14:19:19.090 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-06T14:19:19.090 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.max_total_wal_size: 0 2024-09-06T14:19:19.090 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-06T14:19:19.090 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-06T14:19:19.090 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-06T14:19:19.090 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-06T14:19:19.090 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.max_open_files: -1 2024-09-06T14:19:19.090 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.bytes_per_sync: 0 2024-09-06T14:19:19.090 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-06T14:19:19.090 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-06T14:19:19.090 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.compaction_readahead_size: 0 2024-09-06T14:19:19.091 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.max_background_flushes: -1 2024-09-06T14:19:19.091 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Compression algorithms supported: 2024-09-06T14:19:19.091 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: kZSTD supported: 0 2024-09-06T14:19:19.091 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: kXpressCompression supported: 0 2024-09-06T14:19:19.091 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: kBZip2Compression supported: 0 2024-09-06T14:19:19.091 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-06T14:19:19.091 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: kLZ4Compression supported: 1 2024-09-06T14:19:19.091 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: kZlibCompression supported: 1 2024-09-06T14:19:19.091 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: kLZ4HCCompression supported: 1 2024-09-06T14:19:19.091 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: kSnappyCompression supported: 1 2024-09-06T14:19:19.091 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-06T14:19:19.091 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-06T14:19:19.091 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-09-06T14:19:19.092 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-06T14:19:19.092 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-06T14:19:19.092 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.merge_operator: 2024-09-06T14:19:19.092 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.compaction_filter: None 2024-09-06T14:19:19.092 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.compaction_filter_factory: None 2024-09-06T14:19:19.092 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.sst_partitioner_factory: None 2024-09-06T14:19:19.092 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-06T14:19:19.092 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-06T14:19:19.092 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x556f51bdd4e0) 2024-09-06T14:19:19.092 INFO:journalctl@ceph.mon.a.smithi142.stdout: cache_index_and_filter_blocks: 1 2024-09-06T14:19:19.092 INFO:journalctl@ceph.mon.a.smithi142.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-06T14:19:19.092 INFO:journalctl@ceph.mon.a.smithi142.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-06T14:19:19.092 INFO:journalctl@ceph.mon.a.smithi142.stdout: pin_top_level_index_and_filter: 1 2024-09-06T14:19:19.092 INFO:journalctl@ceph.mon.a.smithi142.stdout: index_type: 0 2024-09-06T14:19:19.093 INFO:journalctl@ceph.mon.a.smithi142.stdout: data_block_index_type: 0 2024-09-06T14:19:19.093 INFO:journalctl@ceph.mon.a.smithi142.stdout: index_shortening: 1 2024-09-06T14:19:19.093 INFO:journalctl@ceph.mon.a.smithi142.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-06T14:19:19.093 INFO:journalctl@ceph.mon.a.smithi142.stdout: checksum: 4 2024-09-06T14:19:19.093 INFO:journalctl@ceph.mon.a.smithi142.stdout: no_block_cache: 0 2024-09-06T14:19:19.093 INFO:journalctl@ceph.mon.a.smithi142.stdout: block_cache: 0x556f51bd0850 2024-09-06T14:19:19.093 INFO:journalctl@ceph.mon.a.smithi142.stdout: block_cache_name: BinnedLRUCache 2024-09-06T14:19:19.093 INFO:journalctl@ceph.mon.a.smithi142.stdout: block_cache_options: 2024-09-06T14:19:19.093 INFO:journalctl@ceph.mon.a.smithi142.stdout: capacity : 536870912 2024-09-06T14:19:19.093 INFO:journalctl@ceph.mon.a.smithi142.stdout: num_shard_bits : 4 2024-09-06T14:19:19.093 INFO:journalctl@ceph.mon.a.smithi142.stdout: strict_capacity_limit : 0 2024-09-06T14:19:19.093 INFO:journalctl@ceph.mon.a.smithi142.stdout: high_pri_pool_ratio: 0.000 2024-09-06T14:19:19.093 INFO:journalctl@ceph.mon.a.smithi142.stdout: block_cache_compressed: (nil) 2024-09-06T14:19:19.094 INFO:journalctl@ceph.mon.a.smithi142.stdout: persistent_cache: (nil) 2024-09-06T14:19:19.094 INFO:journalctl@ceph.mon.a.smithi142.stdout: block_size: 4096 2024-09-06T14:19:19.094 INFO:journalctl@ceph.mon.a.smithi142.stdout: block_size_deviation: 10 2024-09-06T14:19:19.094 INFO:journalctl@ceph.mon.a.smithi142.stdout: block_restart_interval: 16 2024-09-06T14:19:19.094 INFO:journalctl@ceph.mon.a.smithi142.stdout: index_block_restart_interval: 1 2024-09-06T14:19:19.094 INFO:journalctl@ceph.mon.a.smithi142.stdout: metadata_block_size: 4096 2024-09-06T14:19:19.094 INFO:journalctl@ceph.mon.a.smithi142.stdout: partition_filters: 0 2024-09-06T14:19:19.094 INFO:journalctl@ceph.mon.a.smithi142.stdout: use_delta_encoding: 1 2024-09-06T14:19:19.094 INFO:journalctl@ceph.mon.a.smithi142.stdout: filter_policy: bloomfilter 2024-09-06T14:19:19.094 INFO:journalctl@ceph.mon.a.smithi142.stdout: whole_key_filtering: 1 2024-09-06T14:19:19.094 INFO:journalctl@ceph.mon.a.smithi142.stdout: verify_compression: 0 2024-09-06T14:19:19.094 INFO:journalctl@ceph.mon.a.smithi142.stdout: read_amp_bytes_per_bit: 0 2024-09-06T14:19:19.094 INFO:journalctl@ceph.mon.a.smithi142.stdout: format_version: 5 2024-09-06T14:19:19.094 INFO:journalctl@ceph.mon.a.smithi142.stdout: enable_index_compression: 1 2024-09-06T14:19:19.095 INFO:journalctl@ceph.mon.a.smithi142.stdout: block_align: 0 2024-09-06T14:19:19.095 INFO:journalctl@ceph.mon.a.smithi142.stdout: max_auto_readahead_size: 262144 2024-09-06T14:19:19.095 INFO:journalctl@ceph.mon.a.smithi142.stdout: prepopulate_block_cache: 0 2024-09-06T14:19:19.095 INFO:journalctl@ceph.mon.a.smithi142.stdout: initial_auto_readahead_size: 8192 2024-09-06T14:19:19.095 INFO:journalctl@ceph.mon.a.smithi142.stdout: num_file_reads_for_auto_readahead: 2 2024-09-06T14:19:19.095 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.write_buffer_size: 33554432 2024-09-06T14:19:19.095 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.max_write_buffer_number: 2 2024-09-06T14:19:19.095 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.compression: NoCompression 2024-09-06T14:19:19.095 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.bottommost_compression: Disabled 2024-09-06T14:19:19.095 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.prefix_extractor: nullptr 2024-09-06T14:19:19.096 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-06T14:19:19.096 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.num_levels: 7 2024-09-06T14:19:19.096 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-06T14:19:19.096 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-06T14:19:19.096 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-06T14:19:19.096 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-06T14:19:19.096 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-06T14:19:19.096 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-06T14:19:19.096 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-06T14:19:19.096 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-06T14:19:19.096 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-06T14:19:19.096 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-06T14:19:19.097 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-06T14:19:19.097 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-06T14:19:19.097 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-06T14:19:19.097 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.compression_opts.level: 32767 2024-09-06T14:19:19.097 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.compression_opts.strategy: 0 2024-09-06T14:19:19.097 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-06T14:19:19.097 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-06T14:19:19.097 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-06T14:19:19.097 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-06T14:19:19.097 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.compression_opts.enabled: false 2024-09-06T14:19:19.097 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-06T14:19:19.097 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-06T14:19:19.098 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-06T14:19:19.098 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-06T14:19:19.098 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.target_file_size_base: 67108864 2024-09-06T14:19:19.098 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-06T14:19:19.098 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-06T14:19:19.098 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-06T14:19:19.098 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-06T14:19:19.098 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-06T14:19:19.098 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-06T14:19:19.098 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-06T14:19:19.098 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-06T14:19:19.099 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-06T14:19:19.099 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-06T14:19:19.099 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-06T14:19:19.099 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-06T14:19:19.099 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-06T14:19:19.099 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-06T14:19:19.099 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.arena_block_size: 1048576 2024-09-06T14:19:19.099 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-06T14:19:19.099 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-06T14:19:19.099 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.disable_auto_compactions: 0 2024-09-06T14:19:19.099 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-06T14:19:19.100 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-06T14:19:19.100 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-06T14:19:19.100 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-06T14:19:19.100 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-06T14:19:19.100 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-06T14:19:19.100 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-06T14:19:19.100 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-06T14:19:19.100 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-06T14:19:19.100 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-06T14:19:19.100 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-06T14:19:19.101 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.inplace_update_support: 0 2024-09-06T14:19:19.101 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-06T14:19:19.101 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-06T14:19:19.101 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-06T14:19:19.101 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-06T14:19:19.101 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.bloom_locality: 0 2024-09-06T14:19:19.101 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.max_successive_merges: 0 2024-09-06T14:19:19.101 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-06T14:19:19.101 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.paranoid_file_checks: 0 2024-09-06T14:19:19.101 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.force_consistency_checks: 1 2024-09-06T14:19:19.101 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.report_bg_io_stats: 0 2024-09-06T14:19:19.101 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.ttl: 2592000 2024-09-06T14:19:19.101 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-06T14:19:19.101 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-06T14:19:19.102 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-06T14:19:19.102 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.enable_blob_files: false 2024-09-06T14:19:19.102 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.min_blob_size: 0 2024-09-06T14:19:19.102 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.blob_file_size: 268435456 2024-09-06T14:19:19.102 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-06T14:19:19.102 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-06T14:19:19.102 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-06T14:19:19.102 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-06T14:19:19.102 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-06T14:19:19.102 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.blob_file_starting_level: 0 2024-09-06T14:19:19.102 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-06T14:19:19.102 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-09-06T14:19:19.102 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-09-06T14:19:19.103 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 69238300-ce22-4b77-bebb-1b61f5b7ed73 2024-09-06T14:19:19.103 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: EVENT_LOG_v1 {"time_micros": 1725632358825142, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-09-06T14:19:19.103 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-09-06T14:19:19.103 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: EVENT_LOG_v1 {"time_micros": 1725632358825987, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 88349, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 273, "table_properties": {"data_size": 86417, "index_size": 252, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 645, "raw_key_size": 10932, "raw_average_key_size": 47, "raw_value_size": 80235, "raw_average_value_size": 345, "num_data_blocks": 11, "num_entries": 232, "num_filter_entries": 232, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1725632358, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "69238300-ce22-4b77-bebb-1b61f5b7ed73", "db_session_id": "QPZ2N7JIH0C1PW4C03JI", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-09-06T14:19:19.103 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: EVENT_LOG_v1 {"time_micros": 1725632358826081, "job": 1, "event": "recovery_finished"} 2024-09-06T14:19:19.103 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-09-06T14:19:19.103 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-06T14:19:19.103 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x556f51c00e00 2024-09-06T14:19:19.103 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: DB pointer 0x556f51d00000 2024-09-06T14:19:19.103 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-06T14:19:19.103 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-06T14:19:19.103 INFO:journalctl@ceph.mon.a.smithi142.stdout: ** DB Stats ** 2024-09-06T14:19:19.103 INFO:journalctl@ceph.mon.a.smithi142.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-06T14:19:19.103 INFO:journalctl@ceph.mon.a.smithi142.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-06T14:19:19.104 INFO:journalctl@ceph.mon.a.smithi142.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-06T14:19:19.104 INFO:journalctl@ceph.mon.a.smithi142.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-06T14:19:19.104 INFO:journalctl@ceph.mon.a.smithi142.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-06T14:19:19.104 INFO:journalctl@ceph.mon.a.smithi142.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-06T14:19:19.104 INFO:journalctl@ceph.mon.a.smithi142.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-06T14:19:19.104 INFO:journalctl@ceph.mon.a.smithi142.stdout: 2024-09-06T14:19:19.104 INFO:journalctl@ceph.mon.a.smithi142.stdout: ** Compaction Stats [default] ** 2024-09-06T14:19:19.105 INFO:journalctl@ceph.mon.a.smithi142.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-06T14:19:19.105 INFO:journalctl@ceph.mon.a.smithi142.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-06T14:19:19.105 INFO:journalctl@ceph.mon.a.smithi142.stdout: L0 2/0 88.14 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 154.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-06T14:19:19.105 INFO:journalctl@ceph.mon.a.smithi142.stdout: Sum 2/0 88.14 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 154.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-06T14:19:19.105 INFO:journalctl@ceph.mon.a.smithi142.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 154.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-06T14:19:19.105 INFO:journalctl@ceph.mon.a.smithi142.stdout: 2024-09-06T14:19:19.105 INFO:journalctl@ceph.mon.a.smithi142.stdout: ** Compaction Stats [default] ** 2024-09-06T14:19:19.105 INFO:journalctl@ceph.mon.a.smithi142.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-06T14:19:19.105 INFO:journalctl@ceph.mon.a.smithi142.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-06T14:19:19.105 INFO:journalctl@ceph.mon.a.smithi142.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 154.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-06T14:19:19.105 INFO:journalctl@ceph.mon.a.smithi142.stdout: 2024-09-06T14:19:19.105 INFO:journalctl@ceph.mon.a.smithi142.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-06T14:19:19.106 INFO:journalctl@ceph.mon.a.smithi142.stdout: 2024-09-06T14:19:19.106 INFO:journalctl@ceph.mon.a.smithi142.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-06T14:19:19.106 INFO:journalctl@ceph.mon.a.smithi142.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-06T14:19:19.106 INFO:journalctl@ceph.mon.a.smithi142.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-06T14:19:19.106 INFO:journalctl@ceph.mon.a.smithi142.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-06T14:19:19.106 INFO:journalctl@ceph.mon.a.smithi142.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-06T14:19:19.106 INFO:journalctl@ceph.mon.a.smithi142.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-06T14:19:19.106 INFO:journalctl@ceph.mon.a.smithi142.stdout: Cumulative compaction: 0.00 GB write, 27.68 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-06T14:19:19.106 INFO:journalctl@ceph.mon.a.smithi142.stdout: Interval compaction: 0.00 GB write, 27.68 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-06T14:19:19.106 INFO:journalctl@ceph.mon.a.smithi142.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-06T14:19:19.106 INFO:journalctl@ceph.mon.a.smithi142.stdout: Block cache BinnedLRUCache@0x556f51bd0850#2 capacity: 512.00 MB usage: 1.27 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-09-06T14:19:19.106 INFO:journalctl@ceph.mon.a.smithi142.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.83 KB,0.000157952%) IndexBlock(2,0.44 KB,8.34465e-05%) Misc(1,0.00 KB,0%) 2024-09-06T14:19:19.106 INFO:journalctl@ceph.mon.a.smithi142.stdout: 2024-09-06T14:19:19.106 INFO:journalctl@ceph.mon.a.smithi142.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-06T14:19:19.107 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: starting mon.a rank 0 at public addrs [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] at bind addrs [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid ce32462e-6c5a-11ef-bce4-c7b262605968 2024-09-06T14:19:19.107 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: mon.a@-1(???) e1 preinit fsid ce32462e-6c5a-11ef-bce4-c7b262605968 2024-09-06T14:19:19.107 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: mon.a@-1(???).mds e1 new map 2024-09-06T14:19:19.107 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: mon.a@-1(???).mds e1 print_map 2024-09-06T14:19:19.107 INFO:journalctl@ceph.mon.a.smithi142.stdout: e1 2024-09-06T14:19:19.107 INFO:journalctl@ceph.mon.a.smithi142.stdout: btime 2024-09-06T14:19:13:103085+0000 2024-09-06T14:19:19.107 INFO:journalctl@ceph.mon.a.smithi142.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-06T14:19:19.107 INFO:journalctl@ceph.mon.a.smithi142.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-06T14:19:19.107 INFO:journalctl@ceph.mon.a.smithi142.stdout: legacy client fscid: -1 2024-09-06T14:19:19.107 INFO:journalctl@ceph.mon.a.smithi142.stdout: 2024-09-06T14:19:19.107 INFO:journalctl@ceph.mon.a.smithi142.stdout: No filesystems configured 2024-09-06T14:19:19.107 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-06T14:19:19.107 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-06T14:19:19.107 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-06T14:19:19.108 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-06T14:19:19.108 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-09-06T14:19:19.108 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 bash[29939]: e23b8f60e296b972502999b12c5cbedbb4c39b4bb7bc67fe0b8ad0da8551e445 2024-09-06T14:19:19.108 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 systemd[1]: Started Ceph mon.a for ce32462e-6c5a-11ef-bce4-c7b262605968. 2024-09-06T14:19:19.108 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-06T14:19:19.108 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: monmap epoch 1 2024-09-06T14:19:19.108 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: fsid ce32462e-6c5a-11ef-bce4-c7b262605968 2024-09-06T14:19:19.108 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: last_changed 2024-09-06T14:19:01.768092+0000 2024-09-06T14:19:19.108 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: created 2024-09-06T14:19:01.768092+0000 2024-09-06T14:19:19.108 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: min_mon_release 19 (squid) 2024-09-06T14:19:19.108 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: election_strategy: 1 2024-09-06T14:19:19.108 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: 0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-06T14:19:19.108 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: fsmap 2024-09-06T14:19:19.108 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: osdmap e1: 0 total, 0 up, 0 in 2024-09-06T14:19:19.109 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:18 smithi142 ceph-mon[29958]: mgrmap e1: no daemons active 2024-09-06T14:19:19.918 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:19.269+0000 7f850e105640 1 Processor -- start 2024-09-06T14:19:19.918 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:19.269+0000 7f850e105640 1 -- start start 2024-09-06T14:19:19.918 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:19.269+0000 7f850e105640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f85081066f0 0x7f8508106af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:19:19.919 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:19.269+0000 7f850e105640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f85081070e0 con 0x7f85081066f0 2024-09-06T14:19:19.919 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:19.270+0000 7f85077fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f85081066f0 0x7f8508106af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:19:19.919 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:19.270+0000 7f85077fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f85081066f0 0x7f8508106af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:51312/0 (socket says 172.21.15.142:51312) 2024-09-06T14:19:19.919 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:19.270+0000 7f85077fe640 1 -- 172.21.15.142:0/3929122335 learned_addr learned my addr 172.21.15.142:0/3929122335 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:19:19.919 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:19.270+0000 7f85077fe640 1 -- 172.21.15.142:0/3929122335 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8508107260 con 0x7f85081066f0 2024-09-06T14:19:19.919 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:19.272+0000 7f85077fe640 1 --2- 172.21.15.142:0/3929122335 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f85081066f0 0x7f8508106af0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f84f8009820 tx=0x7f84f802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3021fc554df365e3 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:19:19.919 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:19.272+0000 7f85067fc640 1 -- 172.21.15.142:0/3929122335 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f84f8037070 con 0x7f85081066f0 2024-09-06T14:19:19.919 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:19.272+0000 7f85067fc640 1 -- 172.21.15.142:0/3929122335 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1023+0+0 (secure 0 0 0) 0x7f84f80328a0 con 0x7f85081066f0 2024-09-06T14:19:19.919 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:19.272+0000 7f85067fc640 1 -- 172.21.15.142:0/3929122335 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f84f8030690 con 0x7f85081066f0 2024-09-06T14:19:19.919 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:19.273+0000 7f850e105640 1 -- 172.21.15.142:0/3929122335 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f85081066f0 msgr2=0x7f8508106af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:19:19.919 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:19.273+0000 7f850e105640 1 --2- 172.21.15.142:0/3929122335 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f85081066f0 0x7f8508106af0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f84f8009820 tx=0x7f84f802a270 comp rx=0 tx=0).stop 2024-09-06T14:19:19.919 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:19.273+0000 7f850e105640 1 -- 172.21.15.142:0/3929122335 shutdown_connections 2024-09-06T14:19:19.920 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:19.273+0000 7f850e105640 1 --2- 172.21.15.142:0/3929122335 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f85081066f0 0x7f8508106af0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:19:19.920 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:19.273+0000 7f850e105640 1 -- 172.21.15.142:0/3929122335 >> 172.21.15.142:0/3929122335 conn(0x7f8508102020 msgr2=0x7f8508104440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:19:19.920 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:19.273+0000 7f850e105640 1 -- 172.21.15.142:0/3929122335 shutdown_connections 2024-09-06T14:19:19.920 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:19.273+0000 7f850e105640 1 -- 172.21.15.142:0/3929122335 wait complete. 2024-09-06T14:19:19.920 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:19.273+0000 7f850e105640 1 Processor -- start 2024-09-06T14:19:19.920 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:19.274+0000 7f850e105640 1 -- start start 2024-09-06T14:19:19.920 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:19.274+0000 7f850e105640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f85081066f0 0x7f850807c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:19:19.920 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:19.274+0000 7f850e105640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f850810ab20 con 0x7f85081066f0 2024-09-06T14:19:19.920 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:19.274+0000 7f85077fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f85081066f0 0x7f850807c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:19:19.920 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:19.274+0000 7f85077fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f85081066f0 0x7f850807c200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:51324/0 (socket says 172.21.15.142:51324) 2024-09-06T14:19:19.920 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:19.274+0000 7f85077fe640 1 -- 172.21.15.142:0/261267422 learned_addr learned my addr 172.21.15.142:0/261267422 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:19:19.920 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:19.274+0000 7f85077fe640 1 -- 172.21.15.142:0/261267422 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f850807c760 con 0x7f85081066f0 2024-09-06T14:19:19.921 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:19.274+0000 7f85077fe640 1 --2- 172.21.15.142:0/261267422 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f85081066f0 0x7f850807c200 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f84f8006f10 tx=0x7f84f8032670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:19:19.921 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:19.274+0000 7f8504ff9640 1 -- 172.21.15.142:0/261267422 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f84f8037040 con 0x7f85081066f0 2024-09-06T14:19:19.921 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:19.275+0000 7f8504ff9640 1 -- 172.21.15.142:0/261267422 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1023+0+0 (secure 0 0 0) 0x7f84f8040070 con 0x7f85081066f0 2024-09-06T14:19:19.921 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:19.275+0000 7f850e105640 1 -- 172.21.15.142:0/261267422 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f850807ab50 con 0x7f85081066f0 2024-09-06T14:19:19.921 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:19.275+0000 7f8504ff9640 1 -- 172.21.15.142:0/261267422 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f84f803a390 con 0x7f85081066f0 2024-09-06T14:19:19.921 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:19.275+0000 7f850e105640 1 -- 172.21.15.142:0/261267422 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f850807b0d0 con 0x7f85081066f0 2024-09-06T14:19:19.921 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:19.275+0000 7f8504ff9640 1 -- 172.21.15.142:0/261267422 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f84f803a600 con 0x7f85081066f0 2024-09-06T14:19:19.921 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:19.275+0000 7f8504ff9640 1 -- 172.21.15.142:0/261267422 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f84f8045780 con 0x7f85081066f0 2024-09-06T14:19:19.921 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:19.276+0000 7f850e105640 1 -- 172.21.15.142:0/261267422 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8508106b70 con 0x7f85081066f0 2024-09-06T14:19:19.921 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:19.278+0000 7f8504ff9640 1 -- 172.21.15.142:0/261267422 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f84f8032830 con 0x7f85081066f0 2024-09-06T14:19:19.921 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:19.336+0000 7f850e105640 1 -- 172.21.15.142:0/261267422 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7f850807b660 con 0x7f85081066f0 2024-09-06T14:19:19.921 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:19.340+0000 7f8504ff9640 1 -- 172.21.15.142:0/261267422 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7f84f803b430 con 0x7f85081066f0 2024-09-06T14:19:19.921 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:19.341+0000 7f850e105640 1 -- 172.21.15.142:0/261267422 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f85081066f0 msgr2=0x7f850807c200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:19:19.922 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:19.341+0000 7f850e105640 1 --2- 172.21.15.142:0/261267422 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f85081066f0 0x7f850807c200 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f84f8006f10 tx=0x7f84f8032670 comp rx=0 tx=0).stop 2024-09-06T14:19:19.922 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:19.341+0000 7f850e105640 1 -- 172.21.15.142:0/261267422 shutdown_connections 2024-09-06T14:19:19.922 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:19.341+0000 7f850e105640 1 --2- 172.21.15.142:0/261267422 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f85081066f0 0x7f850807c200 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:19:19.922 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:19.341+0000 7f850e105640 1 -- 172.21.15.142:0/261267422 >> 172.21.15.142:0/261267422 conn(0x7f8508102020 msgr2=0x7f8508102b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:19:19.922 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:19.341+0000 7f850e105640 1 -- 172.21.15.142:0/261267422 shutdown_connections 2024-09-06T14:19:19.922 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:19.341+0000 7f850e105640 1 -- 172.21.15.142:0/261267422 wait complete. 2024-09-06T14:19:19.922 INFO:teuthology.orchestra.run.smithi142.stdout:Wrote config to /etc/ceph/ceph.conf 2024-09-06T14:19:19.936 INFO:teuthology.orchestra.run.smithi142.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-09-06T14:19:19.936 INFO:teuthology.orchestra.run.smithi142.stdout:Creating mgr... 2024-09-06T14:19:19.936 INFO:teuthology.orchestra.run.smithi142.stdout:Verifying port 0.0.0.0:9283 ... 2024-09-06T14:19:19.937 INFO:teuthology.orchestra.run.smithi142.stdout:Verifying port 0.0.0.0:8765 ... 2024-09-06T14:19:20.167 INFO:teuthology.orchestra.run.smithi142.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-ce32462e-6c5a-11ef-bce4-c7b262605968@mgr.y 2024-09-06T14:19:20.167 INFO:teuthology.orchestra.run.smithi142.stdout:systemctl: stderr Failed to reset failed state of unit ceph-ce32462e-6c5a-11ef-bce4-c7b262605968@mgr.y.service: Unit ceph-ce32462e-6c5a-11ef-bce4-c7b262605968@mgr.y.service not loaded. 2024-09-06T14:19:20.323 INFO:teuthology.orchestra.run.smithi142.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-ce32462e-6c5a-11ef-bce4-c7b262605968.target.wants/ceph-ce32462e-6c5a-11ef-bce4-c7b262605968@mgr.y.service → /etc/systemd/system/ceph-ce32462e-6c5a-11ef-bce4-c7b262605968@.service. 2024-09-06T14:19:20.456 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:20 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/261267422' entity='client.admin' 2024-09-06T14:19:20.457 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:19:20 smithi142 systemd[1]: Starting Ceph mgr.y for ce32462e-6c5a-11ef-bce4-c7b262605968... 2024-09-06T14:19:20.830 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:19:20 smithi142 podman[30169]: 2024-09-06 14:19:20.464376127 +0000 UTC m=+0.015712119 image pull 1c45aca0610a312233aa2613f7b416b2884ae14779f1297041485ab28f1c8110 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 2024-09-06T14:19:20.830 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:19:20 smithi142 podman[30169]: 2024-09-06 14:19:20.836251567 +0000 UTC m=+0.387587550 container create 3f6dd6c6ef80d91011b70aa7fbb1be61721ed05bebda3c56771f65f6d4465ae8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-athakkar-testing-2024-09-04-1837-8b38c33, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.37.2) 2024-09-06T14:19:21.099 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:19:21 smithi142 podman[30169]: 2024-09-06 14:19:21.038542101 +0000 UTC m=+0.589878097 container init 3f6dd6c6ef80d91011b70aa7fbb1be61721ed05bebda3c56771f65f6d4465ae8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-athakkar-testing-2024-09-04-1837-8b38c33, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-06T14:19:21.099 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:19:21 smithi142 podman[30169]: 2024-09-06 14:19:21.041440019 +0000 UTC m=+0.592776005 container start 3f6dd6c6ef80d91011b70aa7fbb1be61721ed05bebda3c56771f65f6d4465ae8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.37.2, RELEASE=wip-athakkar-testing-2024-09-04-1837-8b38c33, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-06T14:19:21.113 INFO:teuthology.orchestra.run.smithi142.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-06T14:19:21.113 INFO:teuthology.orchestra.run.smithi142.stdout:systemctl: stdout disabled 2024-09-06T14:19:21.120 INFO:teuthology.orchestra.run.smithi142.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-06T14:19:21.120 INFO:teuthology.orchestra.run.smithi142.stdout:systemctl: stdout inactive 2024-09-06T14:19:21.121 INFO:teuthology.orchestra.run.smithi142.stdout:firewalld.service is not enabled 2024-09-06T14:19:21.121 INFO:teuthology.orchestra.run.smithi142.stdout:Not possible to enable service . firewalld.service is not available 2024-09-06T14:19:21.130 INFO:teuthology.orchestra.run.smithi142.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-06T14:19:21.130 INFO:teuthology.orchestra.run.smithi142.stdout:systemctl: stdout disabled 2024-09-06T14:19:21.137 INFO:teuthology.orchestra.run.smithi142.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-06T14:19:21.138 INFO:teuthology.orchestra.run.smithi142.stdout:systemctl: stdout inactive 2024-09-06T14:19:21.138 INFO:teuthology.orchestra.run.smithi142.stdout:firewalld.service is not enabled 2024-09-06T14:19:21.138 INFO:teuthology.orchestra.run.smithi142.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-09-06T14:19:21.138 INFO:teuthology.orchestra.run.smithi142.stdout:Waiting for mgr to start... 2024-09-06T14:19:21.138 INFO:teuthology.orchestra.run.smithi142.stdout:Waiting for mgr... 2024-09-06T14:19:21.399 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:19:21 smithi142 bash[30169]: 3f6dd6c6ef80d91011b70aa7fbb1be61721ed05bebda3c56771f65f6d4465ae8 2024-09-06T14:19:21.400 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:19:21 smithi142 systemd[1]: Started Ceph mgr.y for ce32462e-6c5a-11ef-bce4-c7b262605968. 2024-09-06T14:19:21.956 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:21 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/1945180511' entity='client.admin' cmd={"prefix": "status", "format": "json-pretty"} : dispatch 2024-09-06T14:19:22.253 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout 2024-09-06T14:19:22.254 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout { 2024-09-06T14:19:22.254 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "fsid": "ce32462e-6c5a-11ef-bce4-c7b262605968", 2024-09-06T14:19:22.254 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "health": { 2024-09-06T14:19:22.254 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-06T14:19:22.254 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-06T14:19:22.254 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-06T14:19:22.254 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout }, 2024-09-06T14:19:22.254 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-06T14:19:22.255 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-06T14:19:22.255 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout 0 2024-09-06T14:19:22.255 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout ], 2024-09-06T14:19:22.255 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-06T14:19:22.255 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "a" 2024-09-06T14:19:22.255 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout ], 2024-09-06T14:19:22.255 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-09-06T14:19:22.255 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-06T14:19:22.256 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-06T14:19:22.256 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-09-06T14:19:22.256 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-06T14:19:22.256 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout }, 2024-09-06T14:19:22.256 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-06T14:19:22.256 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-06T14:19:22.256 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-06T14:19:22.256 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-06T14:19:22.257 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-06T14:19:22.266 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-06T14:19:22.266 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-06T14:19:22.266 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-06T14:19:22.266 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout }, 2024-09-06T14:19:22.266 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-06T14:19:22.266 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-06T14:19:22.266 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-06T14:19:22.266 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-06T14:19:22.267 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-06T14:19:22.267 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-06T14:19:22.267 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-06T14:19:22.267 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-06T14:19:22.267 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-06T14:19:22.267 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout }, 2024-09-06T14:19:22.267 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-06T14:19:22.267 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-06T14:19:22.267 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "btime": "2024-09-06T14:19:13:103085+0000", 2024-09-06T14:19:22.267 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-06T14:19:22.267 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-06T14:19:22.267 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout }, 2024-09-06T14:19:22.267 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-06T14:19:22.267 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-06T14:19:22.268 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-06T14:19:22.268 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-06T14:19:22.268 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-06T14:19:22.268 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-06T14:19:22.268 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "restful" 2024-09-06T14:19:22.268 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout ], 2024-09-06T14:19:22.268 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-06T14:19:22.268 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout }, 2024-09-06T14:19:22.268 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-06T14:19:22.268 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-06T14:19:22.268 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "modified": "2024-09-06T14:19:13.103728+0000", 2024-09-06T14:19:22.269 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-06T14:19:22.269 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout }, 2024-09-06T14:19:22.269 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-06T14:19:22.269 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout } 2024-09-06T14:19:22.269 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:21.589+0000 7f02d92a8640 1 Processor -- start 2024-09-06T14:19:22.269 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:21.589+0000 7f02d92a8640 1 -- start start 2024-09-06T14:19:22.269 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:21.589+0000 7f02d92a8640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f02d4079c40 0x7f02d4109270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:19:22.269 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:21.589+0000 7f02d92a8640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f02d41097b0 con 0x7f02d4079c40 2024-09-06T14:19:22.269 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:21.590+0000 7f02d2ffd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f02d4079c40 0x7f02d4109270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:19:22.269 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:21.590+0000 7f02d2ffd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f02d4079c40 0x7f02d4109270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:51362/0 (socket says 172.21.15.142:51362) 2024-09-06T14:19:22.269 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:21.590+0000 7f02d2ffd640 1 -- 172.21.15.142:0/1977376730 learned_addr learned my addr 172.21.15.142:0/1977376730 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:19:22.269 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:21.590+0000 7f02d2ffd640 1 -- 172.21.15.142:0/1977376730 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02d4109930 con 0x7f02d4079c40 2024-09-06T14:19:22.269 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:21.591+0000 7f02d2ffd640 1 --2- 172.21.15.142:0/1977376730 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f02d4079c40 0x7f02d4109270 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f02c4009820 tx=0x7f02c402a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a24777257abd3b8 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:19:22.270 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:21.591+0000 7f02d27fc640 1 -- 172.21.15.142:0/1977376730 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f02c4037070 con 0x7f02d4079c40 2024-09-06T14:19:22.270 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:21.591+0000 7f02d27fc640 1 -- 172.21.15.142:0/1977376730 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1023+0+0 (secure 0 0 0) 0x7f02c40328a0 con 0x7f02d4079c40 2024-09-06T14:19:22.270 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:21.591+0000 7f02d27fc640 1 -- 172.21.15.142:0/1977376730 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f02c4030710 con 0x7f02d4079c40 2024-09-06T14:19:22.270 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:21.592+0000 7f02d92a8640 1 -- 172.21.15.142:0/1977376730 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f02d4079c40 msgr2=0x7f02d4109270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:19:22.270 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:21.592+0000 7f02d92a8640 1 --2- 172.21.15.142:0/1977376730 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f02d4079c40 0x7f02d4109270 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f02c4009820 tx=0x7f02c402a300 comp rx=0 tx=0).stop 2024-09-06T14:19:22.270 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:21.592+0000 7f02d92a8640 1 -- 172.21.15.142:0/1977376730 shutdown_connections 2024-09-06T14:19:22.270 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:21.592+0000 7f02d92a8640 1 --2- 172.21.15.142:0/1977376730 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f02d4079c40 0x7f02d4109270 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:19:22.270 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:21.592+0000 7f02d92a8640 1 -- 172.21.15.142:0/1977376730 >> 172.21.15.142:0/1977376730 conn(0x7f02d4100f40 msgr2=0x7f02d4103360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:19:22.270 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:21.592+0000 7f02d92a8640 1 -- 172.21.15.142:0/1977376730 shutdown_connections 2024-09-06T14:19:22.270 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:21.592+0000 7f02d92a8640 1 -- 172.21.15.142:0/1977376730 wait complete. 2024-09-06T14:19:22.270 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:21.593+0000 7f02d92a8640 1 Processor -- start 2024-09-06T14:19:22.270 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:21.593+0000 7f02d92a8640 1 -- start start 2024-09-06T14:19:22.270 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:21.593+0000 7f02d92a8640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f02d4079c40 0x7f02d419a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:19:22.271 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:21.593+0000 7f02d92a8640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f02d410a9f0 con 0x7f02d4079c40 2024-09-06T14:19:22.271 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:21.593+0000 7f02d2ffd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f02d4079c40 0x7f02d419a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:19:22.271 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:21.593+0000 7f02d2ffd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f02d4079c40 0x7f02d419a6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:51370/0 (socket says 172.21.15.142:51370) 2024-09-06T14:19:22.271 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:21.593+0000 7f02d2ffd640 1 -- 172.21.15.142:0/1945180511 learned_addr learned my addr 172.21.15.142:0/1945180511 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:19:22.271 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:21.593+0000 7f02d2ffd640 1 -- 172.21.15.142:0/1945180511 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02d419ac10 con 0x7f02d4079c40 2024-09-06T14:19:22.271 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:21.593+0000 7f02d2ffd640 1 --2- 172.21.15.142:0/1945180511 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f02d4079c40 0x7f02d419a6b0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f02c4030460 tx=0x7f02c4030710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:19:22.271 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:21.594+0000 7f02d0ff9640 1 -- 172.21.15.142:0/1945180511 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f02c4037040 con 0x7f02d4079c40 2024-09-06T14:19:22.271 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:21.594+0000 7f02d0ff9640 1 -- 172.21.15.142:0/1945180511 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1023+0+0 (secure 0 0 0) 0x7f02c4040070 con 0x7f02d4079c40 2024-09-06T14:19:22.271 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:21.594+0000 7f02d92a8640 1 -- 172.21.15.142:0/1945180511 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02d419aea0 con 0x7f02d4079c40 2024-09-06T14:19:22.271 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:21.594+0000 7f02d0ff9640 1 -- 172.21.15.142:0/1945180511 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f02c403be30 con 0x7f02d4079c40 2024-09-06T14:19:22.271 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:21.594+0000 7f02d92a8640 1 -- 172.21.15.142:0/1945180511 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02d419b860 con 0x7f02d4079c40 2024-09-06T14:19:22.271 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:21.594+0000 7f02d0ff9640 1 -- 172.21.15.142:0/1945180511 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f02c4045460 con 0x7f02d4079c40 2024-09-06T14:19:22.271 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:21.594+0000 7f02d0ff9640 1 -- 172.21.15.142:0/1945180511 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f02c4044720 con 0x7f02d4079c40 2024-09-06T14:19:22.272 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:21.595+0000 7f02d92a8640 1 -- 172.21.15.142:0/1945180511 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02d41072e0 con 0x7f02d4079c40 2024-09-06T14:19:22.272 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:21.597+0000 7f02d0ff9640 1 -- 172.21.15.142:0/1945180511 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f02c403b5c0 con 0x7f02d4079c40 2024-09-06T14:19:22.272 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:21.662+0000 7f02d92a8640 1 -- 172.21.15.142:0/1945180511 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f02d419be10 con 0x7f02d4079c40 2024-09-06T14:19:22.272 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:21.662+0000 7f02d0ff9640 1 -- 172.21.15.142:0/1945180511 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1291 (secure 0 0 0) 0x7f02c4030af0 con 0x7f02d4079c40 2024-09-06T14:19:22.272 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:21.663+0000 7f02d92a8640 1 -- 172.21.15.142:0/1945180511 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f02d4079c40 msgr2=0x7f02d419a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:19:22.272 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:21.663+0000 7f02d92a8640 1 --2- 172.21.15.142:0/1945180511 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f02d4079c40 0x7f02d419a6b0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f02c4030460 tx=0x7f02c4030710 comp rx=0 tx=0).stop 2024-09-06T14:19:22.272 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:21.663+0000 7f02d92a8640 1 -- 172.21.15.142:0/1945180511 shutdown_connections 2024-09-06T14:19:22.272 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:21.663+0000 7f02d92a8640 1 --2- 172.21.15.142:0/1945180511 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f02d4079c40 0x7f02d419a6b0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:19:22.272 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:21.663+0000 7f02d92a8640 1 -- 172.21.15.142:0/1945180511 >> 172.21.15.142:0/1945180511 conn(0x7f02d4100f40 msgr2=0x7f02d4101ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:19:22.273 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:21.663+0000 7f02d92a8640 1 -- 172.21.15.142:0/1945180511 shutdown_connections 2024-09-06T14:19:22.273 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:21.663+0000 7f02d92a8640 1 -- 172.21.15.142:0/1945180511 wait complete. 2024-09-06T14:19:22.273 INFO:teuthology.orchestra.run.smithi142.stdout:mgr not available, waiting (1/15)... 2024-09-06T14:19:22.707 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:19:22 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: 2024-09-06T14:19:22.293+0000 7f49abcfa040 -1 mgr[py] Module not found: 'volumes' 2024-09-06T14:19:22.707 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:19:22 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: 2024-09-06T14:19:22.293+0000 7f49abcfa040 -1 mgr[py] Traceback (most recent call last): 2024-09-06T14:19:22.707 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:19:22 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: File "/usr/share/ceph/mgr/volumes/__init__.py", line 2, in 2024-09-06T14:19:22.707 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:19:22 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: from .module import Module 2024-09-06T14:19:22.707 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:19:22 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: File "/usr/share/ceph/mgr/volumes/module.py", line 9, in 2024-09-06T14:19:22.707 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:19:22 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: from .fs.volume import VolumeClient 2024-09-06T14:19:22.707 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:19:22 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: File "/usr/share/ceph/mgr/volumes/fs/volume.py", line 12, in 2024-09-06T14:19:22.707 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:19:22 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: from ceph.fs.earmarking import CephFSVolumeEarmarking, EarmarkException # type: ignore 2024-09-06T14:19:22.708 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:19:22 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: ModuleNotFoundError: No module named 'ceph.fs' 2024-09-06T14:19:22.708 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:19:22 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: 2024-09-06T14:19:22.708 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:19:22 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: 2024-09-06T14:19:22.294+0000 7f49abcfa040 -1 mgr[py] Class not found in module 'volumes' 2024-09-06T14:19:22.708 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:19:22 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: 2024-09-06T14:19:22.294+0000 7f49abcfa040 -1 mgr[py] Error loading module 'volumes': (2) No such file or directory 2024-09-06T14:19:24.957 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:24 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/1023349693' entity='client.admin' cmd={"prefix": "status", "format": "json-pretty"} : dispatch 2024-09-06T14:19:25.377 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout 2024-09-06T14:19:25.377 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout { 2024-09-06T14:19:25.377 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "fsid": "ce32462e-6c5a-11ef-bce4-c7b262605968", 2024-09-06T14:19:25.377 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "health": { 2024-09-06T14:19:25.378 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-06T14:19:25.378 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-06T14:19:25.378 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-06T14:19:25.378 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout }, 2024-09-06T14:19:25.378 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-06T14:19:25.378 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-06T14:19:25.378 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout 0 2024-09-06T14:19:25.378 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout ], 2024-09-06T14:19:25.378 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-06T14:19:25.378 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "a" 2024-09-06T14:19:25.378 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout ], 2024-09-06T14:19:25.379 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-09-06T14:19:25.379 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-06T14:19:25.379 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-06T14:19:25.379 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-09-06T14:19:25.379 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-06T14:19:25.379 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout }, 2024-09-06T14:19:25.379 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-06T14:19:25.379 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-06T14:19:25.379 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-06T14:19:25.379 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-06T14:19:25.379 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-06T14:19:25.388 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-06T14:19:25.388 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-06T14:19:25.388 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-06T14:19:25.388 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout }, 2024-09-06T14:19:25.388 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-06T14:19:25.388 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-06T14:19:25.388 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-06T14:19:25.388 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-06T14:19:25.389 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-06T14:19:25.389 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-06T14:19:25.389 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-06T14:19:25.389 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-06T14:19:25.389 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-06T14:19:25.389 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout }, 2024-09-06T14:19:25.389 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-06T14:19:25.389 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-06T14:19:25.389 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "btime": "2024-09-06T14:19:13:103085+0000", 2024-09-06T14:19:25.389 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-06T14:19:25.389 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-06T14:19:25.389 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout }, 2024-09-06T14:19:25.389 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-06T14:19:25.389 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-06T14:19:25.390 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-06T14:19:25.390 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-06T14:19:25.390 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-06T14:19:25.390 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-06T14:19:25.390 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "restful" 2024-09-06T14:19:25.390 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout ], 2024-09-06T14:19:25.390 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-06T14:19:25.390 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout }, 2024-09-06T14:19:25.390 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-06T14:19:25.390 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-06T14:19:25.390 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "modified": "2024-09-06T14:19:13.103728+0000", 2024-09-06T14:19:25.390 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-06T14:19:25.390 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout }, 2024-09-06T14:19:25.390 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-06T14:19:25.391 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout } 2024-09-06T14:19:25.391 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:24.702+0000 7f60896f5640 1 Processor -- start 2024-09-06T14:19:25.391 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:24.702+0000 7f60896f5640 1 -- start start 2024-09-06T14:19:25.391 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:24.703+0000 7f60896f5640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6084106680 0x7f6084106a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:19:25.391 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:24.703+0000 7f60896f5640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f6084107070 con 0x7f6084106680 2024-09-06T14:19:25.391 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:24.703+0000 7f6082ffd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6084106680 0x7f6084106a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:19:25.391 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:24.703+0000 7f6082ffd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6084106680 0x7f6084106a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:51374/0 (socket says 172.21.15.142:51374) 2024-09-06T14:19:25.391 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:24.703+0000 7f6082ffd640 1 -- 172.21.15.142:0/2450835663 learned_addr learned my addr 172.21.15.142:0/2450835663 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:19:25.391 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:24.703+0000 7f6082ffd640 1 -- 172.21.15.142:0/2450835663 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60841071f0 con 0x7f6084106680 2024-09-06T14:19:25.391 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:24.704+0000 7f6082ffd640 1 --2- 172.21.15.142:0/2450835663 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6084106680 0x7f6084106a80 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6074009820 tx=0x7f607402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9677fffae6b09a7 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:19:25.391 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:24.704+0000 7f6081ffb640 1 -- 172.21.15.142:0/2450835663 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6074037070 con 0x7f6084106680 2024-09-06T14:19:25.391 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:24.704+0000 7f6081ffb640 1 -- 172.21.15.142:0/2450835663 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1023+0+0 (secure 0 0 0) 0x7f60740328a0 con 0x7f6084106680 2024-09-06T14:19:25.391 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:24.704+0000 7f6081ffb640 1 -- 172.21.15.142:0/2450835663 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6074030690 con 0x7f6084106680 2024-09-06T14:19:25.391 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:24.705+0000 7f60896f5640 1 -- 172.21.15.142:0/2450835663 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6084106680 msgr2=0x7f6084106a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:19:25.392 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:24.705+0000 7f60896f5640 1 --2- 172.21.15.142:0/2450835663 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6084106680 0x7f6084106a80 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6074009820 tx=0x7f607402a270 comp rx=0 tx=0).stop 2024-09-06T14:19:25.392 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:24.705+0000 7f60896f5640 1 -- 172.21.15.142:0/2450835663 shutdown_connections 2024-09-06T14:19:25.392 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:24.705+0000 7f60896f5640 1 --2- 172.21.15.142:0/2450835663 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6084106680 0x7f6084106a80 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:19:25.392 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:24.705+0000 7f60896f5640 1 -- 172.21.15.142:0/2450835663 >> 172.21.15.142:0/2450835663 conn(0x7f6084101fd0 msgr2=0x7f60841043f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:19:25.392 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:24.705+0000 7f60896f5640 1 -- 172.21.15.142:0/2450835663 shutdown_connections 2024-09-06T14:19:25.392 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:24.705+0000 7f60896f5640 1 -- 172.21.15.142:0/2450835663 wait complete. 2024-09-06T14:19:25.392 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:24.706+0000 7f60896f5640 1 Processor -- start 2024-09-06T14:19:25.392 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:24.706+0000 7f60896f5640 1 -- start start 2024-09-06T14:19:25.392 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:24.706+0000 7f60896f5640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6084106680 0x7f608419c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:19:25.393 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:24.706+0000 7f60896f5640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f608410ab00 con 0x7f6084106680 2024-09-06T14:19:25.393 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:24.706+0000 7f6082ffd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6084106680 0x7f608419c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:19:25.393 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:24.706+0000 7f6082ffd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6084106680 0x7f608419c8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:51376/0 (socket says 172.21.15.142:51376) 2024-09-06T14:19:25.393 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:24.706+0000 7f6082ffd640 1 -- 172.21.15.142:0/1023349693 learned_addr learned my addr 172.21.15.142:0/1023349693 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:19:25.393 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:24.707+0000 7f6082ffd640 1 -- 172.21.15.142:0/1023349693 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f608419ce20 con 0x7f6084106680 2024-09-06T14:19:25.393 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:24.707+0000 7f6082ffd640 1 --2- 172.21.15.142:0/1023349693 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6084106680 0x7f608419c8c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6074000c00 tx=0x7f6074032740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:19:25.393 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:24.707+0000 7f606bfff640 1 -- 172.21.15.142:0/1023349693 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6074037040 con 0x7f6084106680 2024-09-06T14:19:25.393 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:24.707+0000 7f60896f5640 1 -- 172.21.15.142:0/1023349693 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f608419d0b0 con 0x7f6084106680 2024-09-06T14:19:25.393 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:24.707+0000 7f606bfff640 1 -- 172.21.15.142:0/1023349693 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1023+0+0 (secure 0 0 0) 0x7f6074040070 con 0x7f6084106680 2024-09-06T14:19:25.394 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:24.707+0000 7f606bfff640 1 -- 172.21.15.142:0/1023349693 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f607403be40 con 0x7f6084106680 2024-09-06T14:19:25.394 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:24.707+0000 7f60896f5640 1 -- 172.21.15.142:0/1023349693 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f608419da70 con 0x7f6084106680 2024-09-06T14:19:25.394 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:24.708+0000 7f606bfff640 1 -- 172.21.15.142:0/1023349693 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f6074045460 con 0x7f6084106680 2024-09-06T14:19:25.394 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:24.708+0000 7f606bfff640 1 -- 172.21.15.142:0/1023349693 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f607403a750 con 0x7f6084106680 2024-09-06T14:19:25.394 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:24.708+0000 7f60896f5640 1 -- 172.21.15.142:0/1023349693 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6084106b00 con 0x7f6084106680 2024-09-06T14:19:25.394 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:24.710+0000 7f606bfff640 1 -- 172.21.15.142:0/1023349693 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f607403b430 con 0x7f6084106680 2024-09-06T14:19:25.394 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:24.773+0000 7f60896f5640 1 -- 172.21.15.142:0/1023349693 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f608419dcd0 con 0x7f6084106680 2024-09-06T14:19:25.394 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:24.773+0000 7f606bfff640 1 -- 172.21.15.142:0/1023349693 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1291 (secure 0 0 0) 0x7f607403b680 con 0x7f6084106680 2024-09-06T14:19:25.394 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:24.774+0000 7f60896f5640 1 -- 172.21.15.142:0/1023349693 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6084106680 msgr2=0x7f608419c8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:19:25.394 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:24.774+0000 7f60896f5640 1 --2- 172.21.15.142:0/1023349693 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6084106680 0x7f608419c8c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6074000c00 tx=0x7f6074032740 comp rx=0 tx=0).stop 2024-09-06T14:19:25.394 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:24.774+0000 7f60896f5640 1 -- 172.21.15.142:0/1023349693 shutdown_connections 2024-09-06T14:19:25.394 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:24.774+0000 7f60896f5640 1 --2- 172.21.15.142:0/1023349693 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6084106680 0x7f608419c8c0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:19:25.394 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:24.775+0000 7f60896f5640 1 -- 172.21.15.142:0/1023349693 >> 172.21.15.142:0/1023349693 conn(0x7f6084101fd0 msgr2=0x7f6084102a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:19:25.395 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:24.775+0000 7f60896f5640 1 -- 172.21.15.142:0/1023349693 shutdown_connections 2024-09-06T14:19:25.395 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:24.775+0000 7f60896f5640 1 -- 172.21.15.142:0/1023349693 wait complete. 2024-09-06T14:19:25.395 INFO:teuthology.orchestra.run.smithi142.stdout:mgr not available, waiting (2/15)... 2024-09-06T14:19:26.456 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:19:26 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: 2024-09-06T14:19:26.078+0000 7f49abcfa040 -1 log_channel(cluster) log [ERR] : Failed to load ceph-mgr modules: volumes 2024-09-06T14:19:26.457 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:26 smithi142 ceph-mon[29958]: Activating manager daemon y 2024-09-06T14:19:26.457 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:26 smithi142 ceph-mon[29958]: mgrmap e2: y(active, starting, since 0.0033352s) 2024-09-06T14:19:26.457 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:26 smithi142 ceph-mon[29958]: from='mgr.14100 172.21.15.142:0/2357524858' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-09-06T14:19:26.458 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:26 smithi142 ceph-mon[29958]: from='mgr.14100 172.21.15.142:0/2357524858' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-09-06T14:19:26.458 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:26 smithi142 ceph-mon[29958]: from='mgr.14100 172.21.15.142:0/2357524858' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-09-06T14:19:26.458 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:26 smithi142 ceph-mon[29958]: from='mgr.14100 172.21.15.142:0/2357524858' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-06T14:19:26.458 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:26 smithi142 ceph-mon[29958]: from='mgr.14100 172.21.15.142:0/2357524858' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-06T14:19:26.458 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:26 smithi142 ceph-mon[29958]: Manager daemon y is now available 2024-09-06T14:19:26.458 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:26 smithi142 ceph-mon[29958]: from='mgr.14100 172.21.15.142:0/2357524858' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-06T14:19:26.459 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:26 smithi142 ceph-mon[29958]: from='mgr.14100 172.21.15.142:0/2357524858' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-06T14:19:26.459 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:26 smithi142 ceph-mon[29958]: from='mgr.14100 172.21.15.142:0/2357524858' entity='mgr.y' 2024-09-06T14:19:26.459 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:26 smithi142 ceph-mon[29958]: from='mgr.14100 172.21.15.142:0/2357524858' entity='mgr.y' 2024-09-06T14:19:26.459 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:26 smithi142 ceph-mon[29958]: from='mgr.14100 172.21.15.142:0/2357524858' entity='mgr.y' 2024-09-06T14:19:28.456 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:28 smithi142 ceph-mon[29958]: mgrmap e3: y(active, since 1.00694s) 2024-09-06T14:19:28.457 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:28 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/1396649916' entity='client.admin' cmd={"prefix": "status", "format": "json-pretty"} : dispatch 2024-09-06T14:19:28.615 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout 2024-09-06T14:19:28.616 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout { 2024-09-06T14:19:28.616 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "fsid": "ce32462e-6c5a-11ef-bce4-c7b262605968", 2024-09-06T14:19:28.616 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "health": { 2024-09-06T14:19:28.616 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-06T14:19:28.616 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-06T14:19:28.616 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-06T14:19:28.616 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout }, 2024-09-06T14:19:28.616 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-06T14:19:28.616 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-06T14:19:28.616 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout 0 2024-09-06T14:19:28.616 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout ], 2024-09-06T14:19:28.616 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-06T14:19:28.616 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "a" 2024-09-06T14:19:28.617 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout ], 2024-09-06T14:19:28.617 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "quorum_age": 9, 2024-09-06T14:19:28.617 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-06T14:19:28.617 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-06T14:19:28.617 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-09-06T14:19:28.617 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-06T14:19:28.617 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout }, 2024-09-06T14:19:28.617 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-06T14:19:28.617 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-06T14:19:28.617 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-06T14:19:28.617 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-06T14:19:28.617 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-06T14:19:28.627 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-06T14:19:28.627 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-06T14:19:28.627 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-06T14:19:28.627 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout }, 2024-09-06T14:19:28.627 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-06T14:19:28.628 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-06T14:19:28.628 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-06T14:19:28.628 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-06T14:19:28.628 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-06T14:19:28.628 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-06T14:19:28.628 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-06T14:19:28.628 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-06T14:19:28.628 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-06T14:19:28.628 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout }, 2024-09-06T14:19:28.628 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-06T14:19:28.628 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-06T14:19:28.628 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "btime": "2024-09-06T14:19:13:103085+0000", 2024-09-06T14:19:28.628 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-06T14:19:28.629 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-06T14:19:28.629 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout }, 2024-09-06T14:19:28.629 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-06T14:19:28.629 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-06T14:19:28.629 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-06T14:19:28.629 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-06T14:19:28.629 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-06T14:19:28.629 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-06T14:19:28.629 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "restful" 2024-09-06T14:19:28.629 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout ], 2024-09-06T14:19:28.629 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-06T14:19:28.629 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout }, 2024-09-06T14:19:28.629 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-06T14:19:28.629 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-06T14:19:28.630 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "modified": "2024-09-06T14:19:13.103728+0000", 2024-09-06T14:19:28.630 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-06T14:19:28.630 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout }, 2024-09-06T14:19:28.630 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-06T14:19:28.630 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout } 2024-09-06T14:19:28.630 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:27.792+0000 7f49269b9640 1 Processor -- start 2024-09-06T14:19:28.630 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:27.792+0000 7f49269b9640 1 -- start start 2024-09-06T14:19:28.630 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:27.793+0000 7f49269b9640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4920106770 0x7f4920106b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:19:28.630 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:27.793+0000 7f49269b9640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f4920107160 con 0x7f4920106770 2024-09-06T14:19:28.630 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:27.793+0000 7f491ffff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4920106770 0x7f4920106b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:19:28.630 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:27.793+0000 7f491ffff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4920106770 0x7f4920106b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:39484/0 (socket says 172.21.15.142:39484) 2024-09-06T14:19:28.630 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:27.793+0000 7f491ffff640 1 -- 172.21.15.142:0/157862230 learned_addr learned my addr 172.21.15.142:0/157862230 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:19:28.630 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:27.793+0000 7f491ffff640 1 -- 172.21.15.142:0/157862230 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49201072e0 con 0x7f4920106770 2024-09-06T14:19:28.631 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:27.794+0000 7f491ffff640 1 --2- 172.21.15.142:0/157862230 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4920106770 0x7f4920106b70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f4910009820 tx=0x7f491002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34c90acd22a4bfb3 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:19:28.631 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:27.794+0000 7f491effd640 1 -- 172.21.15.142:0/157862230 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4910037070 con 0x7f4920106770 2024-09-06T14:19:28.631 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:27.794+0000 7f491effd640 1 -- 172.21.15.142:0/157862230 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1023+0+0 (secure 0 0 0) 0x7f49100328a0 con 0x7f4920106770 2024-09-06T14:19:28.631 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:27.794+0000 7f491effd640 1 -- 172.21.15.142:0/157862230 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4910030690 con 0x7f4920106770 2024-09-06T14:19:28.631 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:27.795+0000 7f49269b9640 1 -- 172.21.15.142:0/157862230 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4920106770 msgr2=0x7f4920106b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:19:28.631 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:27.795+0000 7f49269b9640 1 --2- 172.21.15.142:0/157862230 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4920106770 0x7f4920106b70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f4910009820 tx=0x7f491002a270 comp rx=0 tx=0).stop 2024-09-06T14:19:28.631 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:27.795+0000 7f49269b9640 1 -- 172.21.15.142:0/157862230 shutdown_connections 2024-09-06T14:19:28.631 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:27.795+0000 7f49269b9640 1 --2- 172.21.15.142:0/157862230 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4920106770 0x7f4920106b70 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:19:28.631 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:27.795+0000 7f49269b9640 1 -- 172.21.15.142:0/157862230 >> 172.21.15.142:0/157862230 conn(0x7f4920102060 msgr2=0x7f4920104480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:19:28.631 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:27.795+0000 7f49269b9640 1 -- 172.21.15.142:0/157862230 shutdown_connections 2024-09-06T14:19:28.631 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:27.795+0000 7f49269b9640 1 -- 172.21.15.142:0/157862230 wait complete. 2024-09-06T14:19:28.631 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:27.795+0000 7f49269b9640 1 Processor -- start 2024-09-06T14:19:28.631 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:27.795+0000 7f49269b9640 1 -- start start 2024-09-06T14:19:28.632 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:27.796+0000 7f49269b9640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4920106770 0x7f492019a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:19:28.632 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:27.796+0000 7f49269b9640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f4920108580 con 0x7f4920106770 2024-09-06T14:19:28.632 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:27.796+0000 7f491ffff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4920106770 0x7f492019a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:19:28.632 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:27.796+0000 7f491ffff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4920106770 0x7f492019a6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:39486/0 (socket says 172.21.15.142:39486) 2024-09-06T14:19:28.632 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:27.796+0000 7f491ffff640 1 -- 172.21.15.142:0/1396649916 learned_addr learned my addr 172.21.15.142:0/1396649916 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:19:28.632 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:27.796+0000 7f491ffff640 1 -- 172.21.15.142:0/1396649916 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f492019ac10 con 0x7f4920106770 2024-09-06T14:19:28.632 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:27.796+0000 7f491ffff640 1 --2- 172.21.15.142:0/1396649916 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4920106770 0x7f492019a6b0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f4910006f10 tx=0x7f4910032670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:19:28.632 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:27.797+0000 7f491d7fa640 1 -- 172.21.15.142:0/1396649916 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4910037040 con 0x7f4920106770 2024-09-06T14:19:28.632 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:27.797+0000 7f491d7fa640 1 -- 172.21.15.142:0/1396649916 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1023+0+0 (secure 0 0 0) 0x7f4910040070 con 0x7f4920106770 2024-09-06T14:19:28.632 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:27.797+0000 7f49269b9640 1 -- 172.21.15.142:0/1396649916 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f492019aea0 con 0x7f4920106770 2024-09-06T14:19:28.632 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:27.797+0000 7f49269b9640 1 -- 172.21.15.142:0/1396649916 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f492019b860 con 0x7f4920106770 2024-09-06T14:19:28.632 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:27.797+0000 7f491d7fa640 1 -- 172.21.15.142:0/1396649916 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f491003a390 con 0x7f4920106770 2024-09-06T14:19:28.632 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:27.798+0000 7f491d7fa640 1 -- 172.21.15.142:0/1396649916 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 3) ==== 50009+0+0 (secure 0 0 0) 0x7f491003a600 con 0x7f4920106770 2024-09-06T14:19:28.632 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:27.798+0000 7f491d7fa640 1 --2- 172.21.15.142:0/1396649916 >> v2:172.21.15.142:6800/1202860895 conn(0x7f490803d970 0x7f490803fe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:19:28.633 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:27.798+0000 7f491d7fa640 1 -- 172.21.15.142:0/1396649916 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f4910071820 con 0x7f4920106770 2024-09-06T14:19:28.633 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:27.798+0000 7f491f7fe640 1 --2- 172.21.15.142:0/1396649916 >> v2:172.21.15.142:6800/1202860895 conn(0x7f490803d970 0x7f490803fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:19:28.633 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:27.798+0000 7f49269b9640 1 -- 172.21.15.142:0/1396649916 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f490c0050b0 con 0x7f4920106770 2024-09-06T14:19:28.633 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:27.798+0000 7f491f7fe640 1 --2- 172.21.15.142:0/1396649916 >> v2:172.21.15.142:6800/1202860895 conn(0x7f490803d970 0x7f490803fe30 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f49140098c0 tx=0x7f4914006db0 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:19:28.633 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:27.803+0000 7f491d7fa640 1 -- 172.21.15.142:0/1396649916 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7f491003a8f0 con 0x7f4920106770 2024-09-06T14:19:28.633 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:28.010+0000 7f49269b9640 1 -- 172.21.15.142:0/1396649916 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f490c0053a0 con 0x7f4920106770 2024-09-06T14:19:28.633 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:28.011+0000 7f491d7fa640 1 -- 172.21.15.142:0/1396649916 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1290 (secure 0 0 0) 0x7f4910039d90 con 0x7f4920106770 2024-09-06T14:19:28.633 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:28.012+0000 7f49269b9640 1 -- 172.21.15.142:0/1396649916 >> v2:172.21.15.142:6800/1202860895 conn(0x7f490803d970 msgr2=0x7f490803fe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:19:28.633 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:28.012+0000 7f49269b9640 1 --2- 172.21.15.142:0/1396649916 >> v2:172.21.15.142:6800/1202860895 conn(0x7f490803d970 0x7f490803fe30 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f49140098c0 tx=0x7f4914006db0 comp rx=0 tx=0).stop 2024-09-06T14:19:28.633 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:28.013+0000 7f49269b9640 1 -- 172.21.15.142:0/1396649916 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4920106770 msgr2=0x7f492019a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:19:28.633 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:28.013+0000 7f49269b9640 1 --2- 172.21.15.142:0/1396649916 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4920106770 0x7f492019a6b0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f4910006f10 tx=0x7f4910032670 comp rx=0 tx=0).stop 2024-09-06T14:19:28.633 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:28.013+0000 7f49269b9640 1 -- 172.21.15.142:0/1396649916 shutdown_connections 2024-09-06T14:19:28.633 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:28.013+0000 7f49269b9640 1 --2- 172.21.15.142:0/1396649916 >> v2:172.21.15.142:6800/1202860895 conn(0x7f490803d970 0x7f490803fe30 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:19:28.633 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:28.013+0000 7f49269b9640 1 --2- 172.21.15.142:0/1396649916 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4920106770 0x7f492019a6b0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:19:28.634 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:28.013+0000 7f49269b9640 1 -- 172.21.15.142:0/1396649916 >> 172.21.15.142:0/1396649916 conn(0x7f4920102060 msgr2=0x7f4920102bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:19:28.634 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:28.013+0000 7f49269b9640 1 -- 172.21.15.142:0/1396649916 shutdown_connections 2024-09-06T14:19:28.634 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:28.013+0000 7f49269b9640 1 -- 172.21.15.142:0/1396649916 wait complete. 2024-09-06T14:19:28.634 INFO:teuthology.orchestra.run.smithi142.stdout:mgr is available 2024-09-06T14:19:29.456 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:29 smithi142 ceph-mon[29958]: mgrmap e4: y(active, since 2s) 2024-09-06T14:19:29.745 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout 2024-09-06T14:19:29.787 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout [global] 2024-09-06T14:19:29.787 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout fsid = ce32462e-6c5a-11ef-bce4-c7b262605968 2024-09-06T14:19:29.787 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-09-06T14:19:29.787 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.142:3300,v1:172.21.15.142:6789] 2024-09-06T14:19:29.787 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-06T14:19:29.787 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-06T14:19:29.788 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-06T14:19:29.788 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-06T14:19:29.788 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout 2024-09-06T14:19:29.794 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-06T14:19:29.794 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-06T14:19:29.794 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout 2024-09-06T14:19:29.794 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout [osd] 2024-09-06T14:19:29.794 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-06T14:19:29.794 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-06T14:19:29.794 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:29.004+0000 7fa31cd60640 1 Processor -- start 2024-09-06T14:19:29.795 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:29.004+0000 7fa31cd60640 1 -- start start 2024-09-06T14:19:29.795 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:29.004+0000 7fa31cd60640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa318106710 0x7fa318106b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:19:29.795 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:29.004+0000 7fa31cd60640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fa318107100 con 0x7fa318106710 2024-09-06T14:19:29.795 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:29.004+0000 7fa31659c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa318106710 0x7fa318106b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:19:29.795 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:29.004+0000 7fa31659c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa318106710 0x7fa318106b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:39492/0 (socket says 172.21.15.142:39492) 2024-09-06T14:19:29.795 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:29.004+0000 7fa31659c640 1 -- 172.21.15.142:0/3332527926 learned_addr learned my addr 172.21.15.142:0/3332527926 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:19:29.795 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:29.005+0000 7fa31659c640 1 -- 172.21.15.142:0/3332527926 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa318107280 con 0x7fa318106710 2024-09-06T14:19:29.795 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:29.005+0000 7fa31659c640 1 --2- 172.21.15.142:0/3332527926 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa318106710 0x7fa318106b10 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fa308009820 tx=0x7fa30802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d89d623526c332a6 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:19:29.795 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:29.006+0000 7fa315d9b640 1 -- 172.21.15.142:0/3332527926 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa308037070 con 0x7fa318106710 2024-09-06T14:19:29.795 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:29.006+0000 7fa315d9b640 1 -- 172.21.15.142:0/3332527926 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1023+0+0 (secure 0 0 0) 0x7fa3080328a0 con 0x7fa318106710 2024-09-06T14:19:29.795 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:29.006+0000 7fa315d9b640 1 -- 172.21.15.142:0/3332527926 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa308030690 con 0x7fa318106710 2024-09-06T14:19:29.795 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:29.006+0000 7fa31cd60640 1 -- 172.21.15.142:0/3332527926 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa318106710 msgr2=0x7fa318106b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:19:29.796 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:29.006+0000 7fa31cd60640 1 --2- 172.21.15.142:0/3332527926 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa318106710 0x7fa318106b10 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fa308009820 tx=0x7fa30802a270 comp rx=0 tx=0).stop 2024-09-06T14:19:29.796 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:29.006+0000 7fa31cd60640 1 -- 172.21.15.142:0/3332527926 shutdown_connections 2024-09-06T14:19:29.796 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:29.006+0000 7fa31cd60640 1 --2- 172.21.15.142:0/3332527926 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa318106710 0x7fa318106b10 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:19:29.796 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:29.006+0000 7fa31cd60640 1 -- 172.21.15.142:0/3332527926 >> 172.21.15.142:0/3332527926 conn(0x7fa318102020 msgr2=0x7fa318104440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:19:29.796 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:29.006+0000 7fa31cd60640 1 -- 172.21.15.142:0/3332527926 shutdown_connections 2024-09-06T14:19:29.796 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:29.006+0000 7fa31cd60640 1 -- 172.21.15.142:0/3332527926 wait complete. 2024-09-06T14:19:29.796 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:29.007+0000 7fa31cd60640 1 Processor -- start 2024-09-06T14:19:29.796 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:29.007+0000 7fa31cd60640 1 -- start start 2024-09-06T14:19:29.796 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:29.007+0000 7fa31cd60640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa318106710 0x7fa31819a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:19:29.796 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:29.007+0000 7fa31cd60640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fa31810ab40 con 0x7fa318106710 2024-09-06T14:19:29.796 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:29.007+0000 7fa31659c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa318106710 0x7fa31819a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:19:29.796 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:29.007+0000 7fa31659c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa318106710 0x7fa31819a6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:39508/0 (socket says 172.21.15.142:39508) 2024-09-06T14:19:29.796 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:29.007+0000 7fa31659c640 1 -- 172.21.15.142:0/1585255306 learned_addr learned my addr 172.21.15.142:0/1585255306 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:19:29.797 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:29.007+0000 7fa31659c640 1 -- 172.21.15.142:0/1585255306 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa31819ac40 con 0x7fa318106710 2024-09-06T14:19:29.797 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:29.008+0000 7fa31659c640 1 --2- 172.21.15.142:0/1585255306 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa318106710 0x7fa31819a6e0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fa308006f10 tx=0x7fa308032670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:19:29.797 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:29.008+0000 7fa30f7fe640 1 -- 172.21.15.142:0/1585255306 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa308037040 con 0x7fa318106710 2024-09-06T14:19:29.797 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:29.008+0000 7fa30f7fe640 1 -- 172.21.15.142:0/1585255306 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1023+0+0 (secure 0 0 0) 0x7fa308040070 con 0x7fa318106710 2024-09-06T14:19:29.797 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:29.008+0000 7fa31cd60640 1 -- 172.21.15.142:0/1585255306 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa31819aed0 con 0x7fa318106710 2024-09-06T14:19:29.797 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:29.008+0000 7fa30f7fe640 1 -- 172.21.15.142:0/1585255306 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa30803a390 con 0x7fa318106710 2024-09-06T14:19:29.797 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:29.008+0000 7fa31cd60640 1 -- 172.21.15.142:0/1585255306 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa31819b890 con 0x7fa318106710 2024-09-06T14:19:29.797 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:29.009+0000 7fa30f7fe640 1 -- 172.21.15.142:0/1585255306 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 4) ==== 50064+0+0 (secure 0 0 0) 0x7fa30803a650 con 0x7fa318106710 2024-09-06T14:19:29.797 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:29.009+0000 7fa31cd60640 1 -- 172.21.15.142:0/1585255306 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa318106b90 con 0x7fa318106710 2024-09-06T14:19:29.797 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:29.009+0000 7fa30f7fe640 1 --2- 172.21.15.142:0/1585255306 >> v2:172.21.15.142:6800/1202860895 conn(0x7fa30403d9b0 0x7fa30403fe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:19:29.797 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:29.009+0000 7fa30f7fe640 1 -- 172.21.15.142:0/1585255306 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fa308071b00 con 0x7fa318106710 2024-09-06T14:19:29.797 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:29.009+0000 7fa30ffff640 1 --2- 172.21.15.142:0/1585255306 >> v2:172.21.15.142:6800/1202860895 conn(0x7fa30403d9b0 0x7fa30403fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:19:29.797 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:29.009+0000 7fa30ffff640 1 --2- 172.21.15.142:0/1585255306 >> v2:172.21.15.142:6800/1202860895 conn(0x7fa30403d9b0 0x7fa30403fe70 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fa3000098c0 tx=0x7fa300006db0 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:19:29.798 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:29.013+0000 7fa30f7fe640 1 -- 172.21.15.142:0/1585255306 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7fa30803a940 con 0x7fa318106710 2024-09-06T14:19:29.798 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:29.166+0000 7fa31cd60640 1 -- 172.21.15.142:0/1585255306 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7fa31819bfb0 con 0x7fa318106710 2024-09-06T14:19:29.798 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:29.167+0000 7fa30f7fe640 1 -- 172.21.15.142:0/1585255306 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+376 (secure 0 0 0) 0x7fa308045460 con 0x7fa318106710 2024-09-06T14:19:29.798 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:29.169+0000 7fa31cd60640 1 -- 172.21.15.142:0/1585255306 >> v2:172.21.15.142:6800/1202860895 conn(0x7fa30403d9b0 msgr2=0x7fa30403fe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:19:29.798 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:29.169+0000 7fa31cd60640 1 --2- 172.21.15.142:0/1585255306 >> v2:172.21.15.142:6800/1202860895 conn(0x7fa30403d9b0 0x7fa30403fe70 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fa3000098c0 tx=0x7fa300006db0 comp rx=0 tx=0).stop 2024-09-06T14:19:29.798 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:29.169+0000 7fa31cd60640 1 -- 172.21.15.142:0/1585255306 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa318106710 msgr2=0x7fa31819a6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:19:29.798 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:29.169+0000 7fa31cd60640 1 --2- 172.21.15.142:0/1585255306 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa318106710 0x7fa31819a6e0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fa308006f10 tx=0x7fa308032670 comp rx=0 tx=0).stop 2024-09-06T14:19:29.798 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:29.169+0000 7fa31cd60640 1 -- 172.21.15.142:0/1585255306 shutdown_connections 2024-09-06T14:19:29.799 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:29.169+0000 7fa31cd60640 1 --2- 172.21.15.142:0/1585255306 >> v2:172.21.15.142:6800/1202860895 conn(0x7fa30403d9b0 0x7fa30403fe70 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:19:29.799 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:29.169+0000 7fa31cd60640 1 --2- 172.21.15.142:0/1585255306 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa318106710 0x7fa31819a6e0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:19:29.799 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:29.169+0000 7fa31cd60640 1 -- 172.21.15.142:0/1585255306 >> 172.21.15.142:0/1585255306 conn(0x7fa318102020 msgr2=0x7fa318102b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:19:29.799 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:29.169+0000 7fa31cd60640 1 -- 172.21.15.142:0/1585255306 shutdown_connections 2024-09-06T14:19:29.799 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:29.169+0000 7fa31cd60640 1 -- 172.21.15.142:0/1585255306 wait complete. 2024-09-06T14:19:29.799 INFO:teuthology.orchestra.run.smithi142.stdout:Enabling cephadm module... 2024-09-06T14:19:30.167 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:30 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/1585255306' entity='client.admin' cmd={"prefix": "config assimilate-conf"} : dispatch 2024-09-06T14:19:31.391 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:31 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/402729962' entity='client.admin' cmd={"prefix": "mgr module enable", "module": "cephadm"} : dispatch 2024-09-06T14:19:31.391 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:19:31 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: ignoring --setuser ceph since I am not root 2024-09-06T14:19:31.391 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:19:31 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: ignoring --setgroup ceph since I am not root 2024-09-06T14:19:31.863 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:30.172+0000 7f7112aa7640 1 Processor -- start 2024-09-06T14:19:31.864 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:30.172+0000 7f7112aa7640 1 -- start start 2024-09-06T14:19:31.864 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:30.172+0000 7f7112aa7640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f710c106710 0x7f710c106b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:19:31.864 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:30.172+0000 7f7112aa7640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f710c107100 con 0x7f710c106710 2024-09-06T14:19:31.864 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:30.172+0000 7f7110843640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f710c106710 0x7f710c106b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:19:31.864 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:30.172+0000 7f7110843640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f710c106710 0x7f710c106b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:39512/0 (socket says 172.21.15.142:39512) 2024-09-06T14:19:31.864 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:30.172+0000 7f7110843640 1 -- 172.21.15.142:0/1504138116 learned_addr learned my addr 172.21.15.142:0/1504138116 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:19:31.864 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:30.173+0000 7f7110843640 1 -- 172.21.15.142:0/1504138116 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f710c107280 con 0x7f710c106710 2024-09-06T14:19:31.864 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:30.173+0000 7f7110843640 1 --2- 172.21.15.142:0/1504138116 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f710c106710 0x7f710c106b10 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f7100009820 tx=0x7f710002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fc5090982972a52 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:19:31.864 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:30.173+0000 7f710b7fe640 1 -- 172.21.15.142:0/1504138116 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7100037070 con 0x7f710c106710 2024-09-06T14:19:31.865 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:30.173+0000 7f710b7fe640 1 -- 172.21.15.142:0/1504138116 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1023+0+0 (secure 0 0 0) 0x7f71000328a0 con 0x7f710c106710 2024-09-06T14:19:31.865 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:30.173+0000 7f710b7fe640 1 -- 172.21.15.142:0/1504138116 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7100030690 con 0x7f710c106710 2024-09-06T14:19:31.865 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:30.174+0000 7f7112aa7640 1 -- 172.21.15.142:0/1504138116 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f710c106710 msgr2=0x7f710c106b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:19:31.865 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:30.174+0000 7f7112aa7640 1 --2- 172.21.15.142:0/1504138116 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f710c106710 0x7f710c106b10 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f7100009820 tx=0x7f710002a270 comp rx=0 tx=0).stop 2024-09-06T14:19:31.865 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:30.174+0000 7f7112aa7640 1 -- 172.21.15.142:0/1504138116 shutdown_connections 2024-09-06T14:19:31.865 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:30.174+0000 7f7112aa7640 1 --2- 172.21.15.142:0/1504138116 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f710c106710 0x7f710c106b10 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:19:31.865 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:30.174+0000 7f7112aa7640 1 -- 172.21.15.142:0/1504138116 >> 172.21.15.142:0/1504138116 conn(0x7f710c102020 msgr2=0x7f710c104440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:19:31.865 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:30.174+0000 7f7112aa7640 1 -- 172.21.15.142:0/1504138116 shutdown_connections 2024-09-06T14:19:31.865 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:30.174+0000 7f7112aa7640 1 -- 172.21.15.142:0/1504138116 wait complete. 2024-09-06T14:19:31.865 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:30.175+0000 7f7112aa7640 1 Processor -- start 2024-09-06T14:19:31.865 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:30.175+0000 7f7112aa7640 1 -- start start 2024-09-06T14:19:31.865 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:30.175+0000 7f7112aa7640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f710c106710 0x7f710c19a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:19:31.865 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:30.175+0000 7f7112aa7640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f710c10ab40 con 0x7f710c106710 2024-09-06T14:19:31.866 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:30.175+0000 7f7110843640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f710c106710 0x7f710c19a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:19:31.866 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:30.175+0000 7f7110843640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f710c106710 0x7f710c19a6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:39522/0 (socket says 172.21.15.142:39522) 2024-09-06T14:19:31.866 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:30.175+0000 7f7110843640 1 -- 172.21.15.142:0/402729962 learned_addr learned my addr 172.21.15.142:0/402729962 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:19:31.866 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:30.175+0000 7f7110843640 1 -- 172.21.15.142:0/402729962 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f710c19ac40 con 0x7f710c106710 2024-09-06T14:19:31.866 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:30.175+0000 7f7110843640 1 --2- 172.21.15.142:0/402729962 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f710c106710 0x7f710c19a6e0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f7100000c00 tx=0x7f7100032740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:19:31.866 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:30.176+0000 7f7109ffb640 1 -- 172.21.15.142:0/402729962 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7100037040 con 0x7f710c106710 2024-09-06T14:19:31.866 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:30.176+0000 7f7109ffb640 1 -- 172.21.15.142:0/402729962 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1023+0+0 (secure 0 0 0) 0x7f7100040070 con 0x7f710c106710 2024-09-06T14:19:31.866 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:30.176+0000 7f7112aa7640 1 -- 172.21.15.142:0/402729962 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f710c19aed0 con 0x7f710c106710 2024-09-06T14:19:31.866 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:30.176+0000 7f7109ffb640 1 -- 172.21.15.142:0/402729962 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f710003be40 con 0x7f710c106710 2024-09-06T14:19:31.866 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:30.176+0000 7f7112aa7640 1 -- 172.21.15.142:0/402729962 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f710c19b890 con 0x7f710c106710 2024-09-06T14:19:31.866 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:30.177+0000 7f7109ffb640 1 -- 172.21.15.142:0/402729962 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 4) ==== 50064+0+0 (secure 0 0 0) 0x7f7100045460 con 0x7f710c106710 2024-09-06T14:19:31.866 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:30.177+0000 7f7109ffb640 1 --2- 172.21.15.142:0/402729962 >> v2:172.21.15.142:6800/1202860895 conn(0x7f70fc03da00 0x7f70fc03fec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:19:31.867 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:30.177+0000 7f7109ffb640 1 -- 172.21.15.142:0/402729962 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f7100072690 con 0x7f710c106710 2024-09-06T14:19:31.867 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:30.177+0000 7f710bfff640 1 --2- 172.21.15.142:0/402729962 >> v2:172.21.15.142:6800/1202860895 conn(0x7f70fc03da00 0x7f70fc03fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:19:31.867 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:30.177+0000 7f7112aa7640 1 -- 172.21.15.142:0/402729962 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f710c106b90 con 0x7f710c106710 2024-09-06T14:19:31.867 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:30.177+0000 7f710bfff640 1 --2- 172.21.15.142:0/402729962 >> v2:172.21.15.142:6800/1202860895 conn(0x7f70fc03da00 0x7f70fc03fec0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f70f80098c0 tx=0x7f70f8006db0 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:19:31.867 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:30.181+0000 7f7109ffb640 1 -- 172.21.15.142:0/402729962 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7f710003b430 con 0x7f710c106710 2024-09-06T14:19:31.867 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:30.381+0000 7f7112aa7640 1 -- 172.21.15.142:0/402729962 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7f710c19be20 con 0x7f710c106710 2024-09-06T14:19:31.867 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:31.091+0000 7f7109ffb640 1 -- 172.21.15.142:0/402729962 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7f710002ed40 con 0x7f710c106710 2024-09-06T14:19:31.867 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:31.092+0000 7f7109ffb640 1 -- 172.21.15.142:0/402729962 <== mon.0 v2:172.21.15.142:3300/0 8 ==== mgrmap(e 5) ==== 50075+0+0 (secure 0 0 0) 0x7f7100045750 con 0x7f710c106710 2024-09-06T14:19:31.867 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:31.094+0000 7f7112aa7640 1 -- 172.21.15.142:0/402729962 >> v2:172.21.15.142:6800/1202860895 conn(0x7f70fc03da00 msgr2=0x7f70fc03fec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:19:31.867 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:31.094+0000 7f7112aa7640 1 --2- 172.21.15.142:0/402729962 >> v2:172.21.15.142:6800/1202860895 conn(0x7f70fc03da00 0x7f70fc03fec0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f70f80098c0 tx=0x7f70f8006db0 comp rx=0 tx=0).stop 2024-09-06T14:19:31.867 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:31.094+0000 7f7112aa7640 1 -- 172.21.15.142:0/402729962 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f710c106710 msgr2=0x7f710c19a6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:19:31.867 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:31.094+0000 7f7112aa7640 1 --2- 172.21.15.142:0/402729962 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f710c106710 0x7f710c19a6e0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f7100000c00 tx=0x7f7100032740 comp rx=0 tx=0).stop 2024-09-06T14:19:31.867 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:31.094+0000 7f7112aa7640 1 -- 172.21.15.142:0/402729962 shutdown_connections 2024-09-06T14:19:31.868 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:31.094+0000 7f7112aa7640 1 --2- 172.21.15.142:0/402729962 >> v2:172.21.15.142:6800/1202860895 conn(0x7f70fc03da00 0x7f70fc03fec0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:19:31.868 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:31.094+0000 7f7112aa7640 1 --2- 172.21.15.142:0/402729962 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f710c106710 0x7f710c19a6e0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:19:31.868 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:31.094+0000 7f7112aa7640 1 -- 172.21.15.142:0/402729962 >> 172.21.15.142:0/402729962 conn(0x7f710c102020 msgr2=0x7f710c102b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:19:31.868 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:31.094+0000 7f7112aa7640 1 -- 172.21.15.142:0/402729962 shutdown_connections 2024-09-06T14:19:31.868 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:31.094+0000 7f7112aa7640 1 -- 172.21.15.142:0/402729962 wait complete. 2024-09-06T14:19:32.201 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:32 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/402729962' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-09-06T14:19:32.201 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:32 smithi142 ceph-mon[29958]: mgrmap e5: y(active, since 5s) 2024-09-06T14:19:32.457 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:19:32 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: 2024-09-06T14:19:32.337+0000 7fef714d1040 -1 mgr[py] Module not found: 'volumes' 2024-09-06T14:19:32.457 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:19:32 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: 2024-09-06T14:19:32.337+0000 7fef714d1040 -1 mgr[py] Traceback (most recent call last): 2024-09-06T14:19:32.457 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:19:32 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: File "/usr/share/ceph/mgr/volumes/__init__.py", line 2, in 2024-09-06T14:19:32.458 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:19:32 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: from .module import Module 2024-09-06T14:19:32.458 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:19:32 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: File "/usr/share/ceph/mgr/volumes/module.py", line 9, in 2024-09-06T14:19:32.458 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:19:32 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: from .fs.volume import VolumeClient 2024-09-06T14:19:32.458 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:19:32 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: File "/usr/share/ceph/mgr/volumes/fs/volume.py", line 12, in 2024-09-06T14:19:32.458 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:19:32 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: from ceph.fs.earmarking import CephFSVolumeEarmarking, EarmarkException # type: ignore 2024-09-06T14:19:32.458 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:19:32 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: ModuleNotFoundError: No module named 'ceph.fs' 2024-09-06T14:19:32.459 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:19:32 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: 2024-09-06T14:19:32.459 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:19:32 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: 2024-09-06T14:19:32.338+0000 7fef714d1040 -1 mgr[py] Class not found in module 'volumes' 2024-09-06T14:19:32.459 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:19:32 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: 2024-09-06T14:19:32.338+0000 7fef714d1040 -1 mgr[py] Error loading module 'volumes': (2) No such file or directory 2024-09-06T14:19:33.087 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout { 2024-09-06T14:19:33.088 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-09-06T14:19:33.088 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-06T14:19:33.088 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-09-06T14:19:33.088 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-06T14:19:33.088 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout } 2024-09-06T14:19:33.088 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:32.283+0000 7fcebec89640 1 Processor -- start 2024-09-06T14:19:33.088 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:32.283+0000 7fcebec89640 1 -- start start 2024-09-06T14:19:33.088 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:32.283+0000 7fcebec89640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fceb807a4b0 0x7fceb8109310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:19:33.088 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:32.284+0000 7fcebec89640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fceb8109850 con 0x7fceb807a4b0 2024-09-06T14:19:33.088 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:32.284+0000 7fcebca25640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fceb807a4b0 0x7fceb8109310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:19:33.088 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:32.284+0000 7fcebca25640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fceb807a4b0 0x7fceb8109310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:39558/0 (socket says 172.21.15.142:39558) 2024-09-06T14:19:33.089 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:32.284+0000 7fcebca25640 1 -- 172.21.15.142:0/1586324327 learned_addr learned my addr 172.21.15.142:0/1586324327 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:19:33.089 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:32.284+0000 7fcebca25640 1 -- 172.21.15.142:0/1586324327 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fceb81099d0 con 0x7fceb807a4b0 2024-09-06T14:19:33.089 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:32.285+0000 7fcebca25640 1 --2- 172.21.15.142:0/1586324327 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fceb807a4b0 0x7fceb8109310 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fceac009820 tx=0x7fceac02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=187a00010144de0c server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:19:33.089 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:32.285+0000 7fceb77fe640 1 -- 172.21.15.142:0/1586324327 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fceac037070 con 0x7fceb807a4b0 2024-09-06T14:19:33.089 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:32.285+0000 7fceb77fe640 1 -- 172.21.15.142:0/1586324327 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1023+0+0 (secure 0 0 0) 0x7fceac0328a0 con 0x7fceb807a4b0 2024-09-06T14:19:33.089 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:32.285+0000 7fceb77fe640 1 -- 172.21.15.142:0/1586324327 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fceac030690 con 0x7fceb807a4b0 2024-09-06T14:19:33.089 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:32.285+0000 7fcebec89640 1 -- 172.21.15.142:0/1586324327 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fceb807a4b0 msgr2=0x7fceb8109310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:19:33.089 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:32.285+0000 7fcebec89640 1 --2- 172.21.15.142:0/1586324327 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fceb807a4b0 0x7fceb8109310 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fceac009820 tx=0x7fceac02a270 comp rx=0 tx=0).stop 2024-09-06T14:19:33.089 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:32.285+0000 7fcebec89640 1 -- 172.21.15.142:0/1586324327 shutdown_connections 2024-09-06T14:19:33.089 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:32.285+0000 7fcebec89640 1 --2- 172.21.15.142:0/1586324327 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fceb807a4b0 0x7fceb8109310 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:19:33.089 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:32.285+0000 7fcebec89640 1 -- 172.21.15.142:0/1586324327 >> 172.21.15.142:0/1586324327 conn(0x7fceb8100f40 msgr2=0x7fceb8103360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:19:33.089 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:32.286+0000 7fcebec89640 1 -- 172.21.15.142:0/1586324327 shutdown_connections 2024-09-06T14:19:33.089 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:32.286+0000 7fcebec89640 1 -- 172.21.15.142:0/1586324327 wait complete. 2024-09-06T14:19:33.089 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:32.286+0000 7fcebec89640 1 Processor -- start 2024-09-06T14:19:33.090 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:32.286+0000 7fcebec89640 1 -- start start 2024-09-06T14:19:33.090 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:32.286+0000 7fcebec89640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fceb807a4b0 0x7fceb8106960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:19:33.090 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:32.286+0000 7fcebec89640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fceb810aa20 con 0x7fceb807a4b0 2024-09-06T14:19:33.090 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:32.286+0000 7fcebca25640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fceb807a4b0 0x7fceb8106960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:19:33.090 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:32.286+0000 7fcebca25640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fceb807a4b0 0x7fceb8106960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:39568/0 (socket says 172.21.15.142:39568) 2024-09-06T14:19:33.090 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:32.286+0000 7fcebca25640 1 -- 172.21.15.142:0/4121916612 learned_addr learned my addr 172.21.15.142:0/4121916612 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:19:33.090 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:32.287+0000 7fcebca25640 1 -- 172.21.15.142:0/4121916612 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fceb8106ec0 con 0x7fceb807a4b0 2024-09-06T14:19:33.090 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:32.287+0000 7fcebca25640 1 --2- 172.21.15.142:0/4121916612 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fceb807a4b0 0x7fceb8106960 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fceac006f10 tx=0x7fceac032670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:19:33.090 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:32.287+0000 7fceb5ffb640 1 -- 172.21.15.142:0/4121916612 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fceac037040 con 0x7fceb807a4b0 2024-09-06T14:19:33.090 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:32.287+0000 7fceb5ffb640 1 -- 172.21.15.142:0/4121916612 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1023+0+0 (secure 0 0 0) 0x7fceac040070 con 0x7fceb807a4b0 2024-09-06T14:19:33.090 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:32.287+0000 7fceb5ffb640 1 -- 172.21.15.142:0/4121916612 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fceac03a390 con 0x7fceb807a4b0 2024-09-06T14:19:33.090 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:32.287+0000 7fcebec89640 1 -- 172.21.15.142:0/4121916612 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fceb81052b0 con 0x7fceb807a4b0 2024-09-06T14:19:33.090 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:32.287+0000 7fcebec89640 1 -- 172.21.15.142:0/4121916612 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fceb8105830 con 0x7fceb807a4b0 2024-09-06T14:19:33.091 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:32.288+0000 7fceb5ffb640 1 -- 172.21.15.142:0/4121916612 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) ==== 50075+0+0 (secure 0 0 0) 0x7fceac03b430 con 0x7fceb807a4b0 2024-09-06T14:19:33.091 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:32.288+0000 7fcebec89640 1 -- 172.21.15.142:0/4121916612 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fceb8107380 con 0x7fceb807a4b0 2024-09-06T14:19:33.091 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:32.289+0000 7fceb5ffb640 1 --2- 172.21.15.142:0/4121916612 >> v2:172.21.15.142:6800/1202860895 conn(0x7fcea803da50 0x7fcea803ff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:19:33.091 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:32.289+0000 7fceb5ffb640 1 -- 172.21.15.142:0/4121916612 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fceac0719a0 con 0x7fceb807a4b0 2024-09-06T14:19:33.091 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:32.289+0000 7fceb7fff640 1 -- 172.21.15.142:0/4121916612 >> v2:172.21.15.142:6800/1202860895 conn(0x7fcea803da50 msgr2=0x7fcea803ff10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:6800/1202860895 2024-09-06T14:19:33.091 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:32.289+0000 7fceb7fff640 1 --2- 172.21.15.142:0/4121916612 >> v2:172.21.15.142:6800/1202860895 conn(0x7fcea803da50 0x7fcea803ff10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-06T14:19:33.091 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:32.293+0000 7fceb5ffb640 1 -- 172.21.15.142:0/4121916612 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7fceac043360 con 0x7fceb807a4b0 2024-09-06T14:19:33.091 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:32.490+0000 7fceb7fff640 1 -- 172.21.15.142:0/4121916612 >> v2:172.21.15.142:6800/1202860895 conn(0x7fcea803da50 msgr2=0x7fcea803ff10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:6800/1202860895 2024-09-06T14:19:33.091 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:32.490+0000 7fceb7fff640 1 --2- 172.21.15.142:0/4121916612 >> v2:172.21.15.142:6800/1202860895 conn(0x7fcea803da50 0x7fcea803ff10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-06T14:19:33.091 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:32.497+0000 7fcebec89640 1 -- 172.21.15.142:0/4121916612 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7fceb8105e00 con 0x7fceb807a4b0 2024-09-06T14:19:33.091 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:32.498+0000 7fceb5ffb640 1 -- 172.21.15.142:0/4121916612 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7fceac02e200 con 0x7fceb807a4b0 2024-09-06T14:19:33.091 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:32.500+0000 7fcebec89640 1 -- 172.21.15.142:0/4121916612 >> v2:172.21.15.142:6800/1202860895 conn(0x7fcea803da50 msgr2=0x7fcea803ff10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-06T14:19:33.091 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:32.500+0000 7fcebec89640 1 --2- 172.21.15.142:0/4121916612 >> v2:172.21.15.142:6800/1202860895 conn(0x7fcea803da50 0x7fcea803ff10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:19:33.092 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:32.500+0000 7fcebec89640 1 -- 172.21.15.142:0/4121916612 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fceb807a4b0 msgr2=0x7fceb8106960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:19:33.092 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:32.500+0000 7fcebec89640 1 --2- 172.21.15.142:0/4121916612 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fceb807a4b0 0x7fceb8106960 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fceac006f10 tx=0x7fceac032670 comp rx=0 tx=0).stop 2024-09-06T14:19:33.092 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:32.500+0000 7fcebec89640 1 -- 172.21.15.142:0/4121916612 shutdown_connections 2024-09-06T14:19:33.092 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:32.500+0000 7fcebec89640 1 --2- 172.21.15.142:0/4121916612 >> v2:172.21.15.142:6800/1202860895 conn(0x7fcea803da50 0x7fcea803ff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:19:33.092 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:32.500+0000 7fcebec89640 1 --2- 172.21.15.142:0/4121916612 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fceb807a4b0 0x7fceb8106960 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:19:33.092 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:32.500+0000 7fcebec89640 1 -- 172.21.15.142:0/4121916612 >> 172.21.15.142:0/4121916612 conn(0x7fceb8100f40 msgr2=0x7fceb8101a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:19:33.092 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:32.500+0000 7fcebec89640 1 -- 172.21.15.142:0/4121916612 shutdown_connections 2024-09-06T14:19:33.092 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:32.500+0000 7fcebec89640 1 -- 172.21.15.142:0/4121916612 wait complete. 2024-09-06T14:19:33.092 INFO:teuthology.orchestra.run.smithi142.stdout:Waiting for the mgr to restart... 2024-09-06T14:19:33.092 INFO:teuthology.orchestra.run.smithi142.stdout:Waiting for mgr epoch 5... 2024-09-06T14:19:33.343 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:33 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/4121916612' entity='client.admin' cmd={"prefix": "mgr stat"} : dispatch 2024-09-06T14:19:36.456 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:19:36 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: 2024-09-06T14:19:36.081+0000 7fef714d1040 -1 log_channel(cluster) log [ERR] : Failed to load ceph-mgr modules: volumes 2024-09-06T14:19:36.457 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:36 smithi142 ceph-mon[29958]: Active manager daemon y restarted 2024-09-06T14:19:36.457 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:36 smithi142 ceph-mon[29958]: Activating manager daemon y 2024-09-06T14:19:36.457 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:36 smithi142 ceph-mon[29958]: osdmap e2: 0 total, 0 up, 0 in 2024-09-06T14:19:36.457 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:36 smithi142 ceph-mon[29958]: mgrmap e6: y(active, starting, since 0.00315953s) 2024-09-06T14:19:36.458 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:36 smithi142 ceph-mon[29958]: from='mgr.14116 172.21.15.142:0/3534094648' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-06T14:19:36.458 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:36 smithi142 ceph-mon[29958]: from='mgr.14116 172.21.15.142:0/3534094648' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-06T14:19:36.458 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:36 smithi142 ceph-mon[29958]: from='mgr.14116 172.21.15.142:0/3534094648' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-09-06T14:19:36.458 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:36 smithi142 ceph-mon[29958]: from='mgr.14116 172.21.15.142:0/3534094648' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-09-06T14:19:36.458 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:36 smithi142 ceph-mon[29958]: from='mgr.14116 172.21.15.142:0/3534094648' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-09-06T14:19:36.458 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:36 smithi142 ceph-mon[29958]: Manager daemon y is now available 2024-09-06T14:19:37.685 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout { 2024-09-06T14:19:37.686 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-09-06T14:19:37.686 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-06T14:19:37.686 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout } 2024-09-06T14:19:37.686 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:33.523+0000 7f10360c8640 1 Processor -- start 2024-09-06T14:19:37.686 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:33.523+0000 7f10360c8640 1 -- start start 2024-09-06T14:19:37.687 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:33.524+0000 7f10360c8640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f10301055f0 0x7f10301059f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:19:37.687 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:33.524+0000 7f10360c8640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f1030105fe0 con 0x7f10301055f0 2024-09-06T14:19:37.687 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:33.524+0000 7f102f7fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f10301055f0 0x7f10301059f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:19:37.687 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:33.524+0000 7f102f7fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f10301055f0 0x7f10301059f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:39570/0 (socket says 172.21.15.142:39570) 2024-09-06T14:19:37.687 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:33.524+0000 7f102f7fe640 1 -- 172.21.15.142:0/4187652976 learned_addr learned my addr 172.21.15.142:0/4187652976 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:19:37.687 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:33.524+0000 7f102f7fe640 1 -- 172.21.15.142:0/4187652976 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1030106160 con 0x7f10301055f0 2024-09-06T14:19:37.688 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:33.525+0000 7f102f7fe640 1 --2- 172.21.15.142:0/4187652976 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f10301055f0 0x7f10301059f0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f1020009820 tx=0x7f102002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c722a531ca8f718f server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:19:37.688 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:33.525+0000 7f102effd640 1 -- 172.21.15.142:0/4187652976 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1020037070 con 0x7f10301055f0 2024-09-06T14:19:37.688 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:33.525+0000 7f102effd640 1 -- 172.21.15.142:0/4187652976 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1023+0+0 (secure 0 0 0) 0x7f10200328a0 con 0x7f10301055f0 2024-09-06T14:19:37.688 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:33.525+0000 7f102effd640 1 -- 172.21.15.142:0/4187652976 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1020030690 con 0x7f10301055f0 2024-09-06T14:19:37.688 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:33.526+0000 7f10360c8640 1 -- 172.21.15.142:0/4187652976 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f10301055f0 msgr2=0x7f10301059f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:19:37.688 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:33.526+0000 7f10360c8640 1 --2- 172.21.15.142:0/4187652976 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f10301055f0 0x7f10301059f0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f1020009820 tx=0x7f102002a270 comp rx=0 tx=0).stop 2024-09-06T14:19:37.689 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:33.526+0000 7f10360c8640 1 -- 172.21.15.142:0/4187652976 shutdown_connections 2024-09-06T14:19:37.689 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:33.526+0000 7f10360c8640 1 --2- 172.21.15.142:0/4187652976 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f10301055f0 0x7f10301059f0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:19:37.689 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:33.526+0000 7f10360c8640 1 -- 172.21.15.142:0/4187652976 >> 172.21.15.142:0/4187652976 conn(0x7f1030100ee0 msgr2=0x7f1030103300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:19:37.689 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:33.526+0000 7f10360c8640 1 -- 172.21.15.142:0/4187652976 shutdown_connections 2024-09-06T14:19:37.689 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:33.526+0000 7f10360c8640 1 -- 172.21.15.142:0/4187652976 wait complete. 2024-09-06T14:19:37.689 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:33.526+0000 7f10360c8640 1 Processor -- start 2024-09-06T14:19:37.690 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:33.526+0000 7f10360c8640 1 -- start start 2024-09-06T14:19:37.690 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:33.527+0000 7f10360c8640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f10301055f0 0x7f10301904e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:19:37.690 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:33.527+0000 7f10360c8640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f1030107450 con 0x7f10301055f0 2024-09-06T14:19:37.690 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:33.527+0000 7f102f7fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f10301055f0 0x7f10301904e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:19:37.690 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:33.527+0000 7f102f7fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f10301055f0 0x7f10301904e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:39584/0 (socket says 172.21.15.142:39584) 2024-09-06T14:19:37.690 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:33.527+0000 7f102f7fe640 1 -- 172.21.15.142:0/271929779 learned_addr learned my addr 172.21.15.142:0/271929779 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:19:37.691 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:33.527+0000 7f102f7fe640 1 -- 172.21.15.142:0/271929779 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1030190a40 con 0x7f10301055f0 2024-09-06T14:19:37.691 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:33.527+0000 7f102f7fe640 1 --2- 172.21.15.142:0/271929779 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f10301055f0 0x7f10301904e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f1020006f10 tx=0x7f1020032670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:19:37.691 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:33.527+0000 7f102d7fa640 1 -- 172.21.15.142:0/271929779 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1020037040 con 0x7f10301055f0 2024-09-06T14:19:37.691 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:33.527+0000 7f102d7fa640 1 -- 172.21.15.142:0/271929779 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1023+0+0 (secure 0 0 0) 0x7f1020040070 con 0x7f10301055f0 2024-09-06T14:19:37.691 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:33.527+0000 7f10360c8640 1 -- 172.21.15.142:0/271929779 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f103018ee30 con 0x7f10301055f0 2024-09-06T14:19:37.691 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:33.527+0000 7f102d7fa640 1 -- 172.21.15.142:0/271929779 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f102003a390 con 0x7f10301055f0 2024-09-06T14:19:37.691 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:33.527+0000 7f10360c8640 1 -- 172.21.15.142:0/271929779 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f103018f3b0 con 0x7f10301055f0 2024-09-06T14:19:37.692 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:33.528+0000 7f102d7fa640 1 -- 172.21.15.142:0/271929779 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) ==== 50075+0+0 (secure 0 0 0) 0x7f102003a600 con 0x7f10301055f0 2024-09-06T14:19:37.692 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:33.528+0000 7f102d7fa640 1 --2- 172.21.15.142:0/271929779 >> v2:172.21.15.142:6800/1202860895 conn(0x7f101c03da50 0x7f101c03ff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:19:37.692 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:33.528+0000 7f102d7fa640 1 -- 172.21.15.142:0/271929779 --> v2:172.21.15.142:6800/1202860895 -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f101c0405e0 con 0x7f101c03da50 2024-09-06T14:19:37.692 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:33.528+0000 7f102d7fa640 1 -- 172.21.15.142:0/271929779 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f10200727a0 con 0x7f10301055f0 2024-09-06T14:19:37.692 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:33.528+0000 7f1027fff640 1 -- 172.21.15.142:0/271929779 >> v2:172.21.15.142:6800/1202860895 conn(0x7f101c03da50 msgr2=0x7f101c03ff10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:6800/1202860895 2024-09-06T14:19:37.692 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:33.528+0000 7f1027fff640 1 --2- 172.21.15.142:0/271929779 >> v2:172.21.15.142:6800/1202860895 conn(0x7f101c03da50 0x7f101c03ff10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-06T14:19:37.692 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:33.729+0000 7f1027fff640 1 -- 172.21.15.142:0/271929779 >> v2:172.21.15.142:6800/1202860895 conn(0x7f101c03da50 msgr2=0x7f101c03ff10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:6800/1202860895 2024-09-06T14:19:37.692 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:33.729+0000 7f1027fff640 1 --2- 172.21.15.142:0/271929779 >> v2:172.21.15.142:6800/1202860895 conn(0x7f101c03da50 0x7f101c03ff10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-06T14:19:37.692 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:34.130+0000 7f1027fff640 1 -- 172.21.15.142:0/271929779 >> v2:172.21.15.142:6800/1202860895 conn(0x7f101c03da50 msgr2=0x7f101c03ff10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:6800/1202860895 2024-09-06T14:19:37.692 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:34.130+0000 7f1027fff640 1 --2- 172.21.15.142:0/271929779 >> v2:172.21.15.142:6800/1202860895 conn(0x7f101c03da50 0x7f101c03ff10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-06T14:19:37.693 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:34.931+0000 7f1027fff640 1 -- 172.21.15.142:0/271929779 >> v2:172.21.15.142:6800/1202860895 conn(0x7f101c03da50 msgr2=0x7f101c03ff10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:6800/1202860895 2024-09-06T14:19:37.693 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:34.931+0000 7f1027fff640 1 --2- 172.21.15.142:0/271929779 >> v2:172.21.15.142:6800/1202860895 conn(0x7f101c03da50 0x7f101c03ff10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-06T14:19:37.693 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:36.084+0000 7f102d7fa640 1 -- 172.21.15.142:0/271929779 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mgrmap(e 6) ==== 49928+0+0 (secure 0 0 0) 0x7f10200716b0 con 0x7f10301055f0 2024-09-06T14:19:37.693 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:36.084+0000 7f102d7fa640 1 -- 172.21.15.142:0/271929779 >> v2:172.21.15.142:6800/1202860895 conn(0x7f101c03da50 msgr2=0x7f101c03ff10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-06T14:19:37.693 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:36.084+0000 7f102d7fa640 1 --2- 172.21.15.142:0/271929779 >> v2:172.21.15.142:6800/1202860895 conn(0x7f101c03da50 0x7f101c03ff10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:19:37.693 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:37.086+0000 7f102d7fa640 1 -- 172.21.15.142:0/271929779 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mgrmap(e 7) ==== 50020+0+0 (secure 0 0 0) 0x7f1020040d10 con 0x7f10301055f0 2024-09-06T14:19:37.693 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:37.086+0000 7f102d7fa640 1 --2- 172.21.15.142:0/271929779 >> v2:172.21.15.142:6800/3892080990 conn(0x7f101c041370 0x7f101c043760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:19:37.693 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:37.086+0000 7f102d7fa640 1 -- 172.21.15.142:0/271929779 --> v2:172.21.15.142:6800/3892080990 -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f101c0405e0 con 0x7f101c041370 2024-09-06T14:19:37.693 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:37.086+0000 7f1027fff640 1 --2- 172.21.15.142:0/271929779 >> v2:172.21.15.142:6800/3892080990 conn(0x7f101c041370 0x7f101c043760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:19:37.693 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:37.087+0000 7f1027fff640 1 --2- 172.21.15.142:0/271929779 >> v2:172.21.15.142:6800/3892080990 conn(0x7f101c041370 0x7f101c043760 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f1018003980 tx=0x7f10180092b0 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:19:37.693 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:37.087+0000 7f102d7fa640 1 -- 172.21.15.142:0/271929779 <== mgr.14116 v2:172.21.15.142:6800/3892080990 1 ==== command_reply(tid 0: 0 ) ==== 8+0+9564 (secure 0 0 0) 0x7f101c0405e0 con 0x7f101c041370 2024-09-06T14:19:37.693 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:37.096+0000 7f10360c8640 1 -- 172.21.15.142:0/271929779 --> v2:172.21.15.142:6800/3892080990 -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f1030105a70 con 0x7f101c041370 2024-09-06T14:19:37.693 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:37.096+0000 7f102d7fa640 1 -- 172.21.15.142:0/271929779 <== mgr.14116 v2:172.21.15.142:6800/3892080990 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7f1030105a70 con 0x7f101c041370 2024-09-06T14:19:37.693 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:37.097+0000 7f10360c8640 1 -- 172.21.15.142:0/271929779 >> v2:172.21.15.142:6800/3892080990 conn(0x7f101c041370 msgr2=0x7f101c043760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:19:37.694 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:37.097+0000 7f10360c8640 1 --2- 172.21.15.142:0/271929779 >> v2:172.21.15.142:6800/3892080990 conn(0x7f101c041370 0x7f101c043760 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f1018003980 tx=0x7f10180092b0 comp rx=0 tx=0).stop 2024-09-06T14:19:37.694 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:37.097+0000 7f10360c8640 1 -- 172.21.15.142:0/271929779 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f10301055f0 msgr2=0x7f10301904e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:19:37.694 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:37.097+0000 7f10360c8640 1 --2- 172.21.15.142:0/271929779 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f10301055f0 0x7f10301904e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f1020006f10 tx=0x7f1020032670 comp rx=0 tx=0).stop 2024-09-06T14:19:37.694 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:37.097+0000 7f10360c8640 1 -- 172.21.15.142:0/271929779 shutdown_connections 2024-09-06T14:19:37.694 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:37.097+0000 7f10360c8640 1 --2- 172.21.15.142:0/271929779 >> v2:172.21.15.142:6800/3892080990 conn(0x7f101c041370 0x7f101c043760 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:19:37.694 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:37.097+0000 7f10360c8640 1 --2- 172.21.15.142:0/271929779 >> v2:172.21.15.142:6800/1202860895 conn(0x7f101c03da50 0x7f101c03ff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:19:37.694 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:37.097+0000 7f10360c8640 1 --2- 172.21.15.142:0/271929779 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f10301055f0 0x7f10301904e0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:19:37.694 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:37.097+0000 7f10360c8640 1 -- 172.21.15.142:0/271929779 >> 172.21.15.142:0/271929779 conn(0x7f1030100ee0 msgr2=0x7f10300798f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:19:37.694 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:37.097+0000 7f10360c8640 1 -- 172.21.15.142:0/271929779 shutdown_connections 2024-09-06T14:19:37.694 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:37.097+0000 7f10360c8640 1 -- 172.21.15.142:0/271929779 wait complete. 2024-09-06T14:19:37.694 INFO:teuthology.orchestra.run.smithi142.stdout:mgr epoch 5 is available 2024-09-06T14:19:37.694 INFO:teuthology.orchestra.run.smithi142.stdout:Setting orchestrator backend to cephadm... 2024-09-06T14:19:38.259 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:38 smithi142 ceph-mon[29958]: from='mgr.14116 172.21.15.142:0/3534094648' entity='mgr.y' 2024-09-06T14:19:38.259 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:38 smithi142 ceph-mon[29958]: from='mgr.14116 172.21.15.142:0/3534094648' entity='mgr.y' 2024-09-06T14:19:38.259 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:38 smithi142 ceph-mon[29958]: Found migration_current of "None". Setting to last migration. 2024-09-06T14:19:38.259 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:38 smithi142 ceph-mon[29958]: from='mgr.14116 172.21.15.142:0/3534094648' entity='mgr.y' 2024-09-06T14:19:38.259 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:38 smithi142 ceph-mon[29958]: from='mgr.14116 172.21.15.142:0/3534094648' entity='mgr.y' 2024-09-06T14:19:38.259 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:38 smithi142 ceph-mon[29958]: from='mgr.14116 172.21.15.142:0/3534094648' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-06T14:19:38.259 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:38 smithi142 ceph-mon[29958]: from='mgr.14116 172.21.15.142:0/3534094648' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-06T14:19:38.259 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:38 smithi142 ceph-mon[29958]: from='client.14120 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-09-06T14:19:38.260 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:38 smithi142 ceph-mon[29958]: mgrmap e7: y(active, since 1.00586s) 2024-09-06T14:19:38.260 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:38 smithi142 ceph-mon[29958]: from='mgr.14116 172.21.15.142:0/3534094648' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-06T14:19:38.260 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:38 smithi142 ceph-mon[29958]: from='mgr.14116 172.21.15.142:0/3534094648' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-06T14:19:38.260 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:38 smithi142 ceph-mon[29958]: from='client.14120 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-09-06T14:19:38.260 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:19:38 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: 2024-09-06T14:19:38.006+0000 7feee8783640 -1 log_channel(cephadm) log [ERR] : [06/Sep/2024:14:19:38] ENGINE Error in HTTPServer.serve 2024-09-06T14:19:38.260 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:19:38 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: Traceback (most recent call last): 2024-09-06T14:19:38.260 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:19:38 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-06T14:19:38.260 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:19:38 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: self._connections.run(self.expiration_interval) 2024-09-06T14:19:38.260 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:19:38 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-06T14:19:38.260 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:19:38 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: self._run(expiration_interval) 2024-09-06T14:19:38.260 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:19:38 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-06T14:19:38.261 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:19:38 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: new_conn = self._from_server_socket(self.server.socket) 2024-09-06T14:19:38.261 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:19:38 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-06T14:19:38.261 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:19:38 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-06T14:19:38.261 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:19:38 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-06T14:19:38.261 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:19:38 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: s = self.context.wrap_socket( 2024-09-06T14:19:38.261 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:19:38 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-06T14:19:38.261 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:19:38 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: return self.sslsocket_class._create( 2024-09-06T14:19:38.261 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:19:38 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-06T14:19:38.261 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:19:38 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: self.do_handshake() 2024-09-06T14:19:38.261 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:19:38 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-06T14:19:38.261 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:19:38 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: self._sslobj.do_handshake() 2024-09-06T14:19:38.261 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:19:38 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-06T14:19:38.261 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:19:38 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: 2024-09-06T14:19:38.849 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:38.088+0000 7fd2c99a5640 1 Processor -- start 2024-09-06T14:19:38.849 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:38.088+0000 7fd2c99a5640 1 -- start start 2024-09-06T14:19:38.849 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:38.088+0000 7fd2c99a5640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd2c41088f0 0x7fd2c4108cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:19:38.849 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:38.088+0000 7fd2c99a5640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fd2c41092e0 con 0x7fd2c41088f0 2024-09-06T14:19:38.849 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:38.088+0000 7fd2c2ffd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd2c41088f0 0x7fd2c4108cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:19:38.849 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:38.088+0000 7fd2c2ffd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd2c41088f0 0x7fd2c4108cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:52482/0 (socket says 172.21.15.142:52482) 2024-09-06T14:19:38.849 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:38.088+0000 7fd2c2ffd640 1 -- 172.21.15.142:0/2038959131 learned_addr learned my addr 172.21.15.142:0/2038959131 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:19:38.849 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:38.088+0000 7fd2c2ffd640 1 -- 172.21.15.142:0/2038959131 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2c4109460 con 0x7fd2c41088f0 2024-09-06T14:19:38.849 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:38.089+0000 7fd2c2ffd640 1 --2- 172.21.15.142:0/2038959131 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd2c41088f0 0x7fd2c4108cf0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fd2bc009e80 tx=0x7fd2bc02a7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59167d13c282c5c3 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:19:38.849 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:38.089+0000 7fd2c1ffb640 1 -- 172.21.15.142:0/2038959131 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd2bc036070 con 0x7fd2c41088f0 2024-09-06T14:19:38.850 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:38.089+0000 7fd2c1ffb640 1 -- 172.21.15.142:0/2038959131 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1023+0+0 (secure 0 0 0) 0x7fd2bc031860 con 0x7fd2c41088f0 2024-09-06T14:19:38.850 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:38.089+0000 7fd2c1ffb640 1 -- 172.21.15.142:0/2038959131 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd2bc007880 con 0x7fd2c41088f0 2024-09-06T14:19:38.850 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:38.090+0000 7fd2c99a5640 1 -- 172.21.15.142:0/2038959131 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd2c41088f0 msgr2=0x7fd2c4108cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:19:38.850 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:38.090+0000 7fd2c99a5640 1 --2- 172.21.15.142:0/2038959131 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd2c41088f0 0x7fd2c4108cf0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fd2bc009e80 tx=0x7fd2bc02a7a0 comp rx=0 tx=0).stop 2024-09-06T14:19:38.850 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:38.090+0000 7fd2c99a5640 1 -- 172.21.15.142:0/2038959131 shutdown_connections 2024-09-06T14:19:38.850 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:38.090+0000 7fd2c99a5640 1 --2- 172.21.15.142:0/2038959131 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd2c41088f0 0x7fd2c4108cf0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:19:38.850 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:38.090+0000 7fd2c99a5640 1 -- 172.21.15.142:0/2038959131 >> 172.21.15.142:0/2038959131 conn(0x7fd2c407c030 msgr2=0x7fd2c407c440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:19:38.850 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:38.090+0000 7fd2c99a5640 1 -- 172.21.15.142:0/2038959131 shutdown_connections 2024-09-06T14:19:38.850 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:38.090+0000 7fd2c99a5640 1 -- 172.21.15.142:0/2038959131 wait complete. 2024-09-06T14:19:38.850 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:38.091+0000 7fd2c99a5640 1 Processor -- start 2024-09-06T14:19:38.850 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:38.091+0000 7fd2c99a5640 1 -- start start 2024-09-06T14:19:38.850 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:38.091+0000 7fd2c99a5640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd2c41088f0 0x7fd2c419c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:19:38.850 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:38.091+0000 7fd2c99a5640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fd2c410cd20 con 0x7fd2c41088f0 2024-09-06T14:19:38.851 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:38.091+0000 7fd2c2ffd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd2c41088f0 0x7fd2c419c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:19:38.851 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:38.091+0000 7fd2c2ffd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd2c41088f0 0x7fd2c419c8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:52498/0 (socket says 172.21.15.142:52498) 2024-09-06T14:19:38.851 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:38.091+0000 7fd2c2ffd640 1 -- 172.21.15.142:0/1481369834 learned_addr learned my addr 172.21.15.142:0/1481369834 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:19:38.851 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:38.091+0000 7fd2c2ffd640 1 -- 172.21.15.142:0/1481369834 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2c419ce20 con 0x7fd2c41088f0 2024-09-06T14:19:38.851 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:38.091+0000 7fd2c2ffd640 1 --2- 172.21.15.142:0/1481369834 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd2c41088f0 0x7fd2c419c8c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd2bc008dc0 tx=0x7fd2bc008df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:19:38.851 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:38.092+0000 7fd2c89a3640 1 -- 172.21.15.142:0/1481369834 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd2bc041020 con 0x7fd2c41088f0 2024-09-06T14:19:38.851 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:38.092+0000 7fd2c89a3640 1 -- 172.21.15.142:0/1481369834 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1023+0+0 (secure 0 0 0) 0x7fd2bc031a10 con 0x7fd2c41088f0 2024-09-06T14:19:38.851 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:38.092+0000 7fd2c99a5640 1 -- 172.21.15.142:0/1481369834 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2c419d0b0 con 0x7fd2c41088f0 2024-09-06T14:19:38.851 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:38.092+0000 7fd2c89a3640 1 -- 172.21.15.142:0/1481369834 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd2bc036040 con 0x7fd2c41088f0 2024-09-06T14:19:38.852 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:38.092+0000 7fd2c99a5640 1 -- 172.21.15.142:0/1481369834 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2c419daf0 con 0x7fd2c41088f0 2024-09-06T14:19:38.852 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:38.093+0000 7fd2c89a3640 1 -- 172.21.15.142:0/1481369834 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 7) ==== 50020+0+0 (secure 0 0 0) 0x7fd2bc044020 con 0x7fd2c41088f0 2024-09-06T14:19:38.852 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:38.093+0000 7fd2c99a5640 1 -- 172.21.15.142:0/1481369834 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2c4108d70 con 0x7fd2c41088f0 2024-09-06T14:19:38.852 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:38.093+0000 7fd2c89a3640 1 --2- 172.21.15.142:0/1481369834 >> v2:172.21.15.142:6800/3892080990 conn(0x7fd2ac03d9c0 0x7fd2ac03fe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:19:38.852 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:38.093+0000 7fd2c89a3640 1 -- 172.21.15.142:0/1481369834 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 854+0+0 (secure 0 0 0) 0x7fd2bc072ef0 con 0x7fd2c41088f0 2024-09-06T14:19:38.852 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:38.093+0000 7fd2c27fc640 1 --2- 172.21.15.142:0/1481369834 >> v2:172.21.15.142:6800/3892080990 conn(0x7fd2ac03d9c0 0x7fd2ac03fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:19:38.852 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:38.093+0000 7fd2c27fc640 1 --2- 172.21.15.142:0/1481369834 >> v2:172.21.15.142:6800/3892080990 conn(0x7fd2ac03d9c0 0x7fd2ac03fe80 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fd2b80098c0 tx=0x7fd2b8006db0 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:19:38.852 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:38.097+0000 7fd2c89a3640 1 -- 172.21.15.142:0/1481369834 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7fd2bc04b020 con 0x7fd2c41088f0 2024-09-06T14:19:38.852 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:38.264+0000 7fd2c99a5640 1 -- 172.21.15.142:0/1481369834 --> v2:172.21.15.142:6800/3892080990 -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7fd2c410b420 con 0x7fd2ac03d9c0 2024-09-06T14:19:38.852 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:38.269+0000 7fd2c89a3640 1 -- 172.21.15.142:0/1481369834 <== mgr.14116 v2:172.21.15.142:6800/3892080990 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fd2c410b420 con 0x7fd2ac03d9c0 2024-09-06T14:19:38.853 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:38.271+0000 7fd2c99a5640 1 -- 172.21.15.142:0/1481369834 >> v2:172.21.15.142:6800/3892080990 conn(0x7fd2ac03d9c0 msgr2=0x7fd2ac03fe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:19:38.853 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:38.271+0000 7fd2c99a5640 1 --2- 172.21.15.142:0/1481369834 >> v2:172.21.15.142:6800/3892080990 conn(0x7fd2ac03d9c0 0x7fd2ac03fe80 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fd2b80098c0 tx=0x7fd2b8006db0 comp rx=0 tx=0).stop 2024-09-06T14:19:38.853 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:38.271+0000 7fd2c99a5640 1 -- 172.21.15.142:0/1481369834 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd2c41088f0 msgr2=0x7fd2c419c8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:19:38.853 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:38.271+0000 7fd2c99a5640 1 --2- 172.21.15.142:0/1481369834 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd2c41088f0 0x7fd2c419c8c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd2bc008dc0 tx=0x7fd2bc008df0 comp rx=0 tx=0).stop 2024-09-06T14:19:38.853 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:38.271+0000 7fd2c99a5640 1 -- 172.21.15.142:0/1481369834 shutdown_connections 2024-09-06T14:19:38.853 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:38.272+0000 7fd2c99a5640 1 --2- 172.21.15.142:0/1481369834 >> v2:172.21.15.142:6800/3892080990 conn(0x7fd2ac03d9c0 0x7fd2ac03fe80 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:19:38.853 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:38.272+0000 7fd2c99a5640 1 --2- 172.21.15.142:0/1481369834 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd2c41088f0 0x7fd2c419c8c0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:19:38.853 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:38.272+0000 7fd2c99a5640 1 -- 172.21.15.142:0/1481369834 >> 172.21.15.142:0/1481369834 conn(0x7fd2c407c030 msgr2=0x7fd2c410ad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:19:38.853 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:38.272+0000 7fd2c99a5640 1 -- 172.21.15.142:0/1481369834 shutdown_connections 2024-09-06T14:19:38.853 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:38.272+0000 7fd2c99a5640 1 -- 172.21.15.142:0/1481369834 wait complete. 2024-09-06T14:19:39.176 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:39 smithi142 ceph-mon[29958]: [06/Sep/2024:14:19:37] ENGINE Bus STARTING 2024-09-06T14:19:39.177 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:39 smithi142 ceph-mon[29958]: [06/Sep/2024:14:19:38] ENGINE Serving on https://172.21.15.142:7150 2024-09-06T14:19:39.177 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:39 smithi142 ceph-mon[29958]: [06/Sep/2024:14:19:38] ENGINE Error in HTTPServer.serve 2024-09-06T14:19:39.177 INFO:journalctl@ceph.mon.a.smithi142.stdout: Traceback (most recent call last): 2024-09-06T14:19:39.177 INFO:journalctl@ceph.mon.a.smithi142.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-06T14:19:39.177 INFO:journalctl@ceph.mon.a.smithi142.stdout: self._connections.run(self.expiration_interval) 2024-09-06T14:19:39.177 INFO:journalctl@ceph.mon.a.smithi142.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-06T14:19:39.177 INFO:journalctl@ceph.mon.a.smithi142.stdout: self._run(expiration_interval) 2024-09-06T14:19:39.177 INFO:journalctl@ceph.mon.a.smithi142.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-06T14:19:39.177 INFO:journalctl@ceph.mon.a.smithi142.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-06T14:19:39.177 INFO:journalctl@ceph.mon.a.smithi142.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-06T14:19:39.178 INFO:journalctl@ceph.mon.a.smithi142.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-06T14:19:39.178 INFO:journalctl@ceph.mon.a.smithi142.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-06T14:19:39.178 INFO:journalctl@ceph.mon.a.smithi142.stdout: s = self.context.wrap_socket( 2024-09-06T14:19:39.178 INFO:journalctl@ceph.mon.a.smithi142.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-06T14:19:39.178 INFO:journalctl@ceph.mon.a.smithi142.stdout: return self.sslsocket_class._create( 2024-09-06T14:19:39.178 INFO:journalctl@ceph.mon.a.smithi142.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-06T14:19:39.178 INFO:journalctl@ceph.mon.a.smithi142.stdout: self.do_handshake() 2024-09-06T14:19:39.178 INFO:journalctl@ceph.mon.a.smithi142.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-06T14:19:39.178 INFO:journalctl@ceph.mon.a.smithi142.stdout: self._sslobj.do_handshake() 2024-09-06T14:19:39.178 INFO:journalctl@ceph.mon.a.smithi142.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-06T14:19:39.178 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:39 smithi142 ceph-mon[29958]: [06/Sep/2024:14:19:38] ENGINE Serving on http://172.21.15.142:8765 2024-09-06T14:19:39.178 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:39 smithi142 ceph-mon[29958]: [06/Sep/2024:14:19:38] ENGINE Bus STARTED 2024-09-06T14:19:39.178 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:39 smithi142 ceph-mon[29958]: from='mgr.14116 172.21.15.142:0/3534094648' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-06T14:19:39.178 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:39 smithi142 ceph-mon[29958]: from='mgr.14116 172.21.15.142:0/3534094648' entity='mgr.y' 2024-09-06T14:19:39.179 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:39 smithi142 ceph-mon[29958]: from='mgr.14116 172.21.15.142:0/3534094648' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-06T14:19:40.004 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout value unchanged 2024-09-06T14:19:40.004 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:39.262+0000 7ff8bc651640 1 Processor -- start 2024-09-06T14:19:40.004 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:39.262+0000 7ff8bc651640 1 -- start start 2024-09-06T14:19:40.004 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:39.262+0000 7ff8bc651640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff8b4106710 0x7ff8b4106b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:19:40.004 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:39.262+0000 7ff8bc651640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7ff8b4107100 con 0x7ff8b4106710 2024-09-06T14:19:40.005 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:39.263+0000 7ff8ba3ed640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff8b4106710 0x7ff8b4106b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:19:40.005 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:39.263+0000 7ff8ba3ed640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff8b4106710 0x7ff8b4106b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:52506/0 (socket says 172.21.15.142:52506) 2024-09-06T14:19:40.005 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:39.263+0000 7ff8ba3ed640 1 -- 172.21.15.142:0/4094832100 learned_addr learned my addr 172.21.15.142:0/4094832100 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:19:40.005 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:39.263+0000 7ff8ba3ed640 1 -- 172.21.15.142:0/4094832100 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8b4107280 con 0x7ff8b4106710 2024-09-06T14:19:40.005 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:39.264+0000 7ff8ba3ed640 1 --2- 172.21.15.142:0/4094832100 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff8b4106710 0x7ff8b4106b10 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7ff8b0009820 tx=0x7ff8b002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83cea4b6d80752ff server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:19:40.005 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:39.264+0000 7ff8b93eb640 1 -- 172.21.15.142:0/4094832100 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff8b0037070 con 0x7ff8b4106710 2024-09-06T14:19:40.005 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:39.264+0000 7ff8b93eb640 1 -- 172.21.15.142:0/4094832100 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1023+0+0 (secure 0 0 0) 0x7ff8b00328a0 con 0x7ff8b4106710 2024-09-06T14:19:40.005 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:39.264+0000 7ff8b93eb640 1 -- 172.21.15.142:0/4094832100 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff8b0030690 con 0x7ff8b4106710 2024-09-06T14:19:40.006 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:39.264+0000 7ff8bc651640 1 -- 172.21.15.142:0/4094832100 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff8b4106710 msgr2=0x7ff8b4106b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:19:40.006 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:39.264+0000 7ff8bc651640 1 --2- 172.21.15.142:0/4094832100 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff8b4106710 0x7ff8b4106b10 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7ff8b0009820 tx=0x7ff8b002a270 comp rx=0 tx=0).stop 2024-09-06T14:19:40.006 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:39.264+0000 7ff8bc651640 1 -- 172.21.15.142:0/4094832100 shutdown_connections 2024-09-06T14:19:40.006 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:39.264+0000 7ff8bc651640 1 --2- 172.21.15.142:0/4094832100 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff8b4106710 0x7ff8b4106b10 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:19:40.006 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:39.264+0000 7ff8bc651640 1 -- 172.21.15.142:0/4094832100 >> 172.21.15.142:0/4094832100 conn(0x7ff8b4102020 msgr2=0x7ff8b4104440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:19:40.006 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:39.264+0000 7ff8bc651640 1 -- 172.21.15.142:0/4094832100 shutdown_connections 2024-09-06T14:19:40.006 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:39.264+0000 7ff8bc651640 1 -- 172.21.15.142:0/4094832100 wait complete. 2024-09-06T14:19:40.006 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:39.265+0000 7ff8bc651640 1 Processor -- start 2024-09-06T14:19:40.006 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:39.265+0000 7ff8bc651640 1 -- start start 2024-09-06T14:19:40.006 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:39.265+0000 7ff8bc651640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff8b4106710 0x7ff8b419a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:19:40.006 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:39.265+0000 7ff8bc651640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7ff8b410ab40 con 0x7ff8b4106710 2024-09-06T14:19:40.006 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:39.265+0000 7ff8ba3ed640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff8b4106710 0x7ff8b419a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:19:40.006 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:39.265+0000 7ff8ba3ed640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff8b4106710 0x7ff8b419a700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:52510/0 (socket says 172.21.15.142:52510) 2024-09-06T14:19:40.006 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:39.265+0000 7ff8ba3ed640 1 -- 172.21.15.142:0/2598931844 learned_addr learned my addr 172.21.15.142:0/2598931844 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:19:40.007 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:39.266+0000 7ff8ba3ed640 1 -- 172.21.15.142:0/2598931844 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8b419ac60 con 0x7ff8b4106710 2024-09-06T14:19:40.007 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:39.266+0000 7ff8ba3ed640 1 --2- 172.21.15.142:0/2598931844 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff8b4106710 0x7ff8b419a700 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7ff8b0000c00 tx=0x7ff8b0032740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:19:40.007 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:39.266+0000 7ff8ab7fe640 1 -- 172.21.15.142:0/2598931844 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff8b0037040 con 0x7ff8b4106710 2024-09-06T14:19:40.007 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:39.266+0000 7ff8bc651640 1 -- 172.21.15.142:0/2598931844 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8b419aef0 con 0x7ff8b4106710 2024-09-06T14:19:40.007 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:39.266+0000 7ff8ab7fe640 1 -- 172.21.15.142:0/2598931844 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1023+0+0 (secure 0 0 0) 0x7ff8b0040070 con 0x7ff8b4106710 2024-09-06T14:19:40.007 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:39.266+0000 7ff8ab7fe640 1 -- 172.21.15.142:0/2598931844 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff8b003be40 con 0x7ff8b4106710 2024-09-06T14:19:40.007 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:39.266+0000 7ff8bc651640 1 -- 172.21.15.142:0/2598931844 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8b419b8b0 con 0x7ff8b4106710 2024-09-06T14:19:40.007 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:39.267+0000 7ff8ab7fe640 1 -- 172.21.15.142:0/2598931844 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 8) ==== 50075+0+0 (secure 0 0 0) 0x7ff8b0045460 con 0x7ff8b4106710 2024-09-06T14:19:40.007 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:39.267+0000 7ff8ab7fe640 1 --2- 172.21.15.142:0/2598931844 >> v2:172.21.15.142:6800/3892080990 conn(0x7ff8a003da50 0x7ff8a003ff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:19:40.007 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:39.267+0000 7ff8bc651640 1 -- 172.21.15.142:0/2598931844 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8b4106b90 con 0x7ff8b4106710 2024-09-06T14:19:40.007 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:39.267+0000 7ff8ab7fe640 1 -- 172.21.15.142:0/2598931844 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 854+0+0 (secure 0 0 0) 0x7ff8b00719a0 con 0x7ff8b4106710 2024-09-06T14:19:40.007 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:39.267+0000 7ff8b9bec640 1 --2- 172.21.15.142:0/2598931844 >> v2:172.21.15.142:6800/3892080990 conn(0x7ff8a003da50 0x7ff8a003ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:19:40.007 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:39.268+0000 7ff8b9bec640 1 --2- 172.21.15.142:0/2598931844 >> v2:172.21.15.142:6800/3892080990 conn(0x7ff8a003da50 0x7ff8a003ff10 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff8a40098c0 tx=0x7ff8a4006db0 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:19:40.008 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:39.271+0000 7ff8ab7fe640 1 -- 172.21.15.142:0/2598931844 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7ff8b003b430 con 0x7ff8b4106710 2024-09-06T14:19:40.008 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:39.426+0000 7ff8bc651640 1 -- 172.21.15.142:0/2598931844 --> v2:172.21.15.142:6800/3892080990 -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7ff8b4103220 con 0x7ff8a003da50 2024-09-06T14:19:40.008 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:39.426+0000 7ff8ab7fe640 1 -- 172.21.15.142:0/2598931844 <== mgr.14116 v2:172.21.15.142:6800/3892080990 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7ff8b4103220 con 0x7ff8a003da50 2024-09-06T14:19:40.008 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:39.428+0000 7ff8bc651640 1 -- 172.21.15.142:0/2598931844 >> v2:172.21.15.142:6800/3892080990 conn(0x7ff8a003da50 msgr2=0x7ff8a003ff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:19:40.008 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:39.428+0000 7ff8bc651640 1 --2- 172.21.15.142:0/2598931844 >> v2:172.21.15.142:6800/3892080990 conn(0x7ff8a003da50 0x7ff8a003ff10 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff8a40098c0 tx=0x7ff8a4006db0 comp rx=0 tx=0).stop 2024-09-06T14:19:40.008 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:39.428+0000 7ff8bc651640 1 -- 172.21.15.142:0/2598931844 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff8b4106710 msgr2=0x7ff8b419a700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:19:40.008 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:39.428+0000 7ff8bc651640 1 --2- 172.21.15.142:0/2598931844 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff8b4106710 0x7ff8b419a700 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7ff8b0000c00 tx=0x7ff8b0032740 comp rx=0 tx=0).stop 2024-09-06T14:19:40.008 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:39.429+0000 7ff8bc651640 1 -- 172.21.15.142:0/2598931844 shutdown_connections 2024-09-06T14:19:40.008 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:39.429+0000 7ff8bc651640 1 --2- 172.21.15.142:0/2598931844 >> v2:172.21.15.142:6800/3892080990 conn(0x7ff8a003da50 0x7ff8a003ff10 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:19:40.008 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:39.429+0000 7ff8bc651640 1 --2- 172.21.15.142:0/2598931844 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff8b4106710 0x7ff8b419a700 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:19:40.009 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:39.429+0000 7ff8bc651640 1 -- 172.21.15.142:0/2598931844 >> 172.21.15.142:0/2598931844 conn(0x7ff8b4102020 msgr2=0x7ff8b4102b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:19:40.009 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:39.429+0000 7ff8bc651640 1 -- 172.21.15.142:0/2598931844 shutdown_connections 2024-09-06T14:19:40.009 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:39.429+0000 7ff8bc651640 1 -- 172.21.15.142:0/2598931844 wait complete. 2024-09-06T14:19:40.009 INFO:teuthology.orchestra.run.smithi142.stdout:Generating ssh key... 2024-09-06T14:19:40.248 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:40 smithi142 ceph-mon[29958]: from='client.14126 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-09-06T14:19:40.249 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:40 smithi142 ceph-mon[29958]: mgrmap e8: y(active, since 2s) 2024-09-06T14:19:41.153 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:41 smithi142 ceph-mon[29958]: from='client.14128 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-09-06T14:19:41.153 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:41 smithi142 ceph-mon[29958]: from='mgr.14116 172.21.15.142:0/3534094648' entity='mgr.y' 2024-09-06T14:19:41.153 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:41 smithi142 ceph-mon[29958]: from='mgr.14116 172.21.15.142:0/3534094648' entity='mgr.y' 2024-09-06T14:19:41.153 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:19:40 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: Generating public/private rsa key pair. 2024-09-06T14:19:41.153 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:19:40 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: Your identification has been saved in /tmp/tmpvx18m840/key 2024-09-06T14:19:41.154 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:19:40 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: Your public key has been saved in /tmp/tmpvx18m840/key.pub 2024-09-06T14:19:41.154 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:19:40 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: The key fingerprint is: 2024-09-06T14:19:41.154 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:19:40 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: SHA256:3lXBacbWBBIrF3jOf7EXdC7QJEi065dK28ftTVZylBc ceph-ce32462e-6c5a-11ef-bce4-c7b262605968 2024-09-06T14:19:41.154 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:19:40 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: The key's randomart image is: 2024-09-06T14:19:41.154 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:19:40 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: +---[RSA 3072]----+ 2024-09-06T14:19:41.154 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:19:40 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: | ooo=*+E.| 2024-09-06T14:19:41.154 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:19:40 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: | o.o=Oo=| 2024-09-06T14:19:41.154 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:19:40 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: | o+o=oo+| 2024-09-06T14:19:41.154 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:19:40 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: | +o..+o| 2024-09-06T14:19:41.154 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:19:40 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: | S . ....*| 2024-09-06T14:19:41.154 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:19:40 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: | . o . ..++| 2024-09-06T14:19:41.154 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:19:40 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: | . + o. o+| 2024-09-06T14:19:41.154 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:19:40 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: | . = ooo| 2024-09-06T14:19:41.154 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:19:40 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: | o .. .o| 2024-09-06T14:19:41.155 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:19:40 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: +----[SHA256]-----+ 2024-09-06T14:19:41.460 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:40.433+0000 7f2e866f2640 1 Processor -- start 2024-09-06T14:19:41.461 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:40.433+0000 7f2e866f2640 1 -- start start 2024-09-06T14:19:41.461 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:40.433+0000 7f2e866f2640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2e8007a430 0x7f2e80109350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:19:41.461 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:40.433+0000 7f2e866f2640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f2e80109890 con 0x7f2e8007a430 2024-09-06T14:19:41.461 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:40.433+0000 7f2e7ffff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2e8007a430 0x7f2e80109350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:19:41.461 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:40.434+0000 7f2e7ffff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2e8007a430 0x7f2e80109350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:52518/0 (socket says 172.21.15.142:52518) 2024-09-06T14:19:41.461 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:40.434+0000 7f2e7ffff640 1 -- 172.21.15.142:0/4119861115 learned_addr learned my addr 172.21.15.142:0/4119861115 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:19:41.461 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:40.434+0000 7f2e7ffff640 1 -- 172.21.15.142:0/4119861115 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e80109a10 con 0x7f2e8007a430 2024-09-06T14:19:41.461 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:40.434+0000 7f2e7ffff640 1 --2- 172.21.15.142:0/4119861115 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2e8007a430 0x7f2e80109350 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f2e70009820 tx=0x7f2e7002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b92a75b94086bf3f server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:19:41.461 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:40.435+0000 7f2e7effd640 1 -- 172.21.15.142:0/4119861115 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2e70037070 con 0x7f2e8007a430 2024-09-06T14:19:41.461 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:40.435+0000 7f2e7effd640 1 -- 172.21.15.142:0/4119861115 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1023+0+0 (secure 0 0 0) 0x7f2e700328a0 con 0x7f2e8007a430 2024-09-06T14:19:41.461 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:40.435+0000 7f2e7effd640 1 -- 172.21.15.142:0/4119861115 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2e70030690 con 0x7f2e8007a430 2024-09-06T14:19:41.461 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:40.435+0000 7f2e866f2640 1 -- 172.21.15.142:0/4119861115 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2e8007a430 msgr2=0x7f2e80109350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:19:41.462 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:40.435+0000 7f2e866f2640 1 --2- 172.21.15.142:0/4119861115 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2e8007a430 0x7f2e80109350 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f2e70009820 tx=0x7f2e7002a270 comp rx=0 tx=0).stop 2024-09-06T14:19:41.462 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:40.435+0000 7f2e866f2640 1 -- 172.21.15.142:0/4119861115 shutdown_connections 2024-09-06T14:19:41.462 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:40.435+0000 7f2e866f2640 1 --2- 172.21.15.142:0/4119861115 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2e8007a430 0x7f2e80109350 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:19:41.462 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:40.435+0000 7f2e866f2640 1 -- 172.21.15.142:0/4119861115 >> 172.21.15.142:0/4119861115 conn(0x7f2e80100f20 msgr2=0x7f2e80103340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:19:41.462 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:40.435+0000 7f2e866f2640 1 -- 172.21.15.142:0/4119861115 shutdown_connections 2024-09-06T14:19:41.462 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:40.435+0000 7f2e866f2640 1 -- 172.21.15.142:0/4119861115 wait complete. 2024-09-06T14:19:41.462 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:40.436+0000 7f2e866f2640 1 Processor -- start 2024-09-06T14:19:41.462 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:40.436+0000 7f2e866f2640 1 -- start start 2024-09-06T14:19:41.462 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:40.436+0000 7f2e866f2640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2e8007a430 0x7f2e8019a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:19:41.462 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:40.436+0000 7f2e866f2640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f2e8010aa20 con 0x7f2e8007a430 2024-09-06T14:19:41.462 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:40.436+0000 7f2e7ffff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2e8007a430 0x7f2e8019a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:19:41.462 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:40.436+0000 7f2e7ffff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2e8007a430 0x7f2e8019a6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:52524/0 (socket says 172.21.15.142:52524) 2024-09-06T14:19:41.462 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:40.436+0000 7f2e7ffff640 1 -- 172.21.15.142:0/507006733 learned_addr learned my addr 172.21.15.142:0/507006733 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:19:41.463 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:40.436+0000 7f2e7ffff640 1 -- 172.21.15.142:0/507006733 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e8019ac50 con 0x7f2e8007a430 2024-09-06T14:19:41.463 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:40.437+0000 7f2e7ffff640 1 --2- 172.21.15.142:0/507006733 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2e8007a430 0x7f2e8019a6f0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f2e70006f10 tx=0x7f2e70032670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:19:41.463 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:40.437+0000 7f2e7d7fa640 1 -- 172.21.15.142:0/507006733 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2e70037040 con 0x7f2e8007a430 2024-09-06T14:19:41.463 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:40.437+0000 7f2e7d7fa640 1 -- 172.21.15.142:0/507006733 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1023+0+0 (secure 0 0 0) 0x7f2e70040070 con 0x7f2e8007a430 2024-09-06T14:19:41.463 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:40.437+0000 7f2e866f2640 1 -- 172.21.15.142:0/507006733 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e8019aee0 con 0x7f2e8007a430 2024-09-06T14:19:41.463 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:40.437+0000 7f2e7d7fa640 1 -- 172.21.15.142:0/507006733 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2e7003a390 con 0x7f2e8007a430 2024-09-06T14:19:41.463 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:40.437+0000 7f2e866f2640 1 -- 172.21.15.142:0/507006733 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e8019b8a0 con 0x7f2e8007a430 2024-09-06T14:19:41.463 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:40.438+0000 7f2e7d7fa640 1 -- 172.21.15.142:0/507006733 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 8) ==== 50075+0+0 (secure 0 0 0) 0x7f2e7003a600 con 0x7f2e8007a430 2024-09-06T14:19:41.463 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:40.438+0000 7f2e866f2640 1 -- 172.21.15.142:0/507006733 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e801073c0 con 0x7f2e8007a430 2024-09-06T14:19:41.463 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:40.438+0000 7f2e7d7fa640 1 --2- 172.21.15.142:0/507006733 >> v2:172.21.15.142:6800/3892080990 conn(0x7f2e6803da50 0x7f2e6803ff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:19:41.463 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:40.438+0000 7f2e7d7fa640 1 -- 172.21.15.142:0/507006733 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 854+0+0 (secure 0 0 0) 0x7f2e70071af0 con 0x7f2e8007a430 2024-09-06T14:19:41.463 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:40.438+0000 7f2e7f7fe640 1 --2- 172.21.15.142:0/507006733 >> v2:172.21.15.142:6800/3892080990 conn(0x7f2e6803da50 0x7f2e6803ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:19:41.463 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:40.438+0000 7f2e7f7fe640 1 --2- 172.21.15.142:0/507006733 >> v2:172.21.15.142:6800/3892080990 conn(0x7f2e6803da50 0x7f2e6803ff10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2e740098c0 tx=0x7f2e74006db0 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:19:41.464 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:40.442+0000 7f2e7d7fa640 1 -- 172.21.15.142:0/507006733 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7f2e7003a8f0 con 0x7f2e8007a430 2024-09-06T14:19:41.464 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:40.595+0000 7f2e866f2640 1 -- 172.21.15.142:0/507006733 --> v2:172.21.15.142:6800/3892080990 -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7f2e80102210 con 0x7f2e6803da50 2024-09-06T14:19:41.464 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:40.838+0000 7f2e7d7fa640 1 -- 172.21.15.142:0/507006733 <== mgr.14116 v2:172.21.15.142:6800/3892080990 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f2e80102210 con 0x7f2e6803da50 2024-09-06T14:19:41.464 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:40.841+0000 7f2e866f2640 1 -- 172.21.15.142:0/507006733 >> v2:172.21.15.142:6800/3892080990 conn(0x7f2e6803da50 msgr2=0x7f2e6803ff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:19:41.464 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:40.841+0000 7f2e866f2640 1 --2- 172.21.15.142:0/507006733 >> v2:172.21.15.142:6800/3892080990 conn(0x7f2e6803da50 0x7f2e6803ff10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2e740098c0 tx=0x7f2e74006db0 comp rx=0 tx=0).stop 2024-09-06T14:19:41.464 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:40.841+0000 7f2e866f2640 1 -- 172.21.15.142:0/507006733 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2e8007a430 msgr2=0x7f2e8019a6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:19:41.464 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:40.841+0000 7f2e866f2640 1 --2- 172.21.15.142:0/507006733 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2e8007a430 0x7f2e8019a6f0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f2e70006f10 tx=0x7f2e70032670 comp rx=0 tx=0).stop 2024-09-06T14:19:41.464 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:40.841+0000 7f2e866f2640 1 -- 172.21.15.142:0/507006733 shutdown_connections 2024-09-06T14:19:41.464 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:40.841+0000 7f2e866f2640 1 --2- 172.21.15.142:0/507006733 >> v2:172.21.15.142:6800/3892080990 conn(0x7f2e6803da50 0x7f2e6803ff10 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:19:41.464 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:40.841+0000 7f2e866f2640 1 --2- 172.21.15.142:0/507006733 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2e8007a430 0x7f2e8019a6f0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:19:41.464 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:40.841+0000 7f2e866f2640 1 -- 172.21.15.142:0/507006733 >> 172.21.15.142:0/507006733 conn(0x7f2e80100f20 msgr2=0x7f2e80101b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:19:41.464 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:40.841+0000 7f2e866f2640 1 -- 172.21.15.142:0/507006733 shutdown_connections 2024-09-06T14:19:41.464 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:40.841+0000 7f2e866f2640 1 -- 172.21.15.142:0/507006733 wait complete. 2024-09-06T14:19:42.350 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:42 smithi142 ceph-mon[29958]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-06T14:19:42.350 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:42 smithi142 ceph-mon[29958]: Generating ssh key... 2024-09-06T14:19:42.635 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCUSzFmAPVYQutDQjVksjZD+8qjmKgJU9tbqlPuy65HiuDiUrrerefQP40TZVyEcrViBYVr81EQArBuusVmqCqGricoPa8W6qCsMmBkIXYCfsX0rzRjCgVJ/DfkPW8ljHm1TzJN/HHSHZTlcTcNu8FdmvzFa4qenA2NhuOwv1RSMGRqDAa3/5MKvMUeFcyZROG2rWW1WNVaAn0+yEUYxruisbuGktxa2okA8Qs4qARnMF5vTIMjADz0/Bp/Ds74KDr/L+eB90Rj2MJu3EADEAQbcgp2OGM39//mutyW2d+wI/XRGIl29tKWv3GiNpq2gyyfHtTZEwVcY4QlNHADia9YwLm0XsH1pwpf9n92I+nwLq6A/Id0reAvLDbLjdCRx6VUapqfW2r0IZ0969jTPpDAoCiR9FyH633ambNPua6pAiLiiuaDlSjPwwAObaU2eamplgyyEEPJKPoCcSiZ/QfM73lzPzMb8rzUV+NL7MMF0rjBgvMYfnvERjhzkla4fFU= ceph-ce32462e-6c5a-11ef-bce4-c7b262605968 2024-09-06T14:19:42.635 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:41.840+0000 7f151f67c640 1 Processor -- start 2024-09-06T14:19:42.636 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:41.840+0000 7f151f67c640 1 -- start start 2024-09-06T14:19:42.636 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:41.840+0000 7f151f67c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f151807a490 0x7f15181092f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:19:42.636 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:41.840+0000 7f151f67c640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f1518109830 con 0x7f151807a490 2024-09-06T14:19:42.636 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:41.840+0000 7f151d418640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f151807a490 0x7f15181092f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:19:42.636 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:41.840+0000 7f151d418640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f151807a490 0x7f15181092f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:52538/0 (socket says 172.21.15.142:52538) 2024-09-06T14:19:42.636 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:41.840+0000 7f151d418640 1 -- 172.21.15.142:0/2635225582 learned_addr learned my addr 172.21.15.142:0/2635225582 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:19:42.636 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:41.840+0000 7f151d418640 1 -- 172.21.15.142:0/2635225582 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15181099b0 con 0x7f151807a490 2024-09-06T14:19:42.636 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:41.841+0000 7f151d418640 1 --2- 172.21.15.142:0/2635225582 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f151807a490 0x7f15181092f0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f150c009820 tx=0x7f150c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=228c2288322e3a0c server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:19:42.636 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:41.841+0000 7f150bfff640 1 -- 172.21.15.142:0/2635225582 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f150c037070 con 0x7f151807a490 2024-09-06T14:19:42.636 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:41.841+0000 7f150bfff640 1 -- 172.21.15.142:0/2635225582 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1023+0+0 (secure 0 0 0) 0x7f150c0328a0 con 0x7f151807a490 2024-09-06T14:19:42.637 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:41.841+0000 7f150bfff640 1 -- 172.21.15.142:0/2635225582 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f150c030690 con 0x7f151807a490 2024-09-06T14:19:42.637 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:41.842+0000 7f151f67c640 1 -- 172.21.15.142:0/2635225582 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f151807a490 msgr2=0x7f15181092f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:19:42.637 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:41.842+0000 7f151f67c640 1 --2- 172.21.15.142:0/2635225582 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f151807a490 0x7f15181092f0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f150c009820 tx=0x7f150c02a270 comp rx=0 tx=0).stop 2024-09-06T14:19:42.637 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:41.842+0000 7f151f67c640 1 -- 172.21.15.142:0/2635225582 shutdown_connections 2024-09-06T14:19:42.637 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:41.842+0000 7f151f67c640 1 --2- 172.21.15.142:0/2635225582 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f151807a490 0x7f15181092f0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:19:42.637 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:41.842+0000 7f151f67c640 1 -- 172.21.15.142:0/2635225582 >> 172.21.15.142:0/2635225582 conn(0x7f1518100f20 msgr2=0x7f1518103340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:19:42.637 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:41.842+0000 7f151f67c640 1 -- 172.21.15.142:0/2635225582 shutdown_connections 2024-09-06T14:19:42.637 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:41.842+0000 7f151f67c640 1 -- 172.21.15.142:0/2635225582 wait complete. 2024-09-06T14:19:42.638 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:41.842+0000 7f151f67c640 1 Processor -- start 2024-09-06T14:19:42.638 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:41.843+0000 7f151f67c640 1 -- start start 2024-09-06T14:19:42.638 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:41.843+0000 7f151f67c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f151807a490 0x7f151819a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:19:42.639 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:41.843+0000 7f151f67c640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f151810aa20 con 0x7f151807a490 2024-09-06T14:19:42.639 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:41.843+0000 7f151d418640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f151807a490 0x7f151819a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:19:42.639 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:41.843+0000 7f151d418640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f151807a490 0x7f151819a720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:52552/0 (socket says 172.21.15.142:52552) 2024-09-06T14:19:42.639 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:41.843+0000 7f151d418640 1 -- 172.21.15.142:0/3456641480 learned_addr learned my addr 172.21.15.142:0/3456641480 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:19:42.639 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:41.843+0000 7f151d418640 1 -- 172.21.15.142:0/3456641480 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f151819ac80 con 0x7f151807a490 2024-09-06T14:19:42.640 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:41.843+0000 7f151d418640 1 --2- 172.21.15.142:0/3456641480 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f151807a490 0x7f151819a720 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f150c006f10 tx=0x7f150c032670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:19:42.640 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:41.843+0000 7f150a7fc640 1 -- 172.21.15.142:0/3456641480 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f150c037040 con 0x7f151807a490 2024-09-06T14:19:42.640 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:41.843+0000 7f150a7fc640 1 -- 172.21.15.142:0/3456641480 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1023+0+0 (secure 0 0 0) 0x7f150c040070 con 0x7f151807a490 2024-09-06T14:19:42.640 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:41.843+0000 7f151f67c640 1 -- 172.21.15.142:0/3456641480 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f151819af10 con 0x7f151807a490 2024-09-06T14:19:42.640 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:41.843+0000 7f150a7fc640 1 -- 172.21.15.142:0/3456641480 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f150c03a390 con 0x7f151807a490 2024-09-06T14:19:42.640 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:41.844+0000 7f151f67c640 1 -- 172.21.15.142:0/3456641480 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f151819b8d0 con 0x7f151807a490 2024-09-06T14:19:42.640 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:41.844+0000 7f150a7fc640 1 -- 172.21.15.142:0/3456641480 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 8) ==== 50075+0+0 (secure 0 0 0) 0x7f150c03b430 con 0x7f151807a490 2024-09-06T14:19:42.640 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:41.844+0000 7f151f67c640 1 -- 172.21.15.142:0/3456641480 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1518107360 con 0x7f151807a490 2024-09-06T14:19:42.640 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:41.845+0000 7f150a7fc640 1 --2- 172.21.15.142:0/3456641480 >> v2:172.21.15.142:6800/3892080990 conn(0x7f150403da50 0x7f150403ff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:19:42.640 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:41.845+0000 7f150a7fc640 1 -- 172.21.15.142:0/3456641480 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 854+0+0 (secure 0 0 0) 0x7f150c071990 con 0x7f151807a490 2024-09-06T14:19:42.640 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:41.845+0000 7f151cc17640 1 --2- 172.21.15.142:0/3456641480 >> v2:172.21.15.142:6800/3892080990 conn(0x7f150403da50 0x7f150403ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:19:42.640 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:41.845+0000 7f151cc17640 1 --2- 172.21.15.142:0/3456641480 >> v2:172.21.15.142:6800/3892080990 conn(0x7f150403da50 0x7f150403ff10 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f15140098c0 tx=0x7f1514006db0 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:19:42.641 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:41.849+0000 7f150a7fc640 1 -- 172.21.15.142:0/3456641480 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7f150c02e200 con 0x7f151807a490 2024-09-06T14:19:42.641 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:42.000+0000 7f151f67c640 1 -- 172.21.15.142:0/3456641480 --> v2:172.21.15.142:6800/3892080990 -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7f151819c120 con 0x7f150403da50 2024-09-06T14:19:42.641 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:42.000+0000 7f150a7fc640 1 -- 172.21.15.142:0/3456641480 <== mgr.14116 v2:172.21.15.142:6800/3892080990 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7f151819c120 con 0x7f150403da50 2024-09-06T14:19:42.641 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:42.002+0000 7f151f67c640 1 -- 172.21.15.142:0/3456641480 >> v2:172.21.15.142:6800/3892080990 conn(0x7f150403da50 msgr2=0x7f150403ff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:19:42.641 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:42.002+0000 7f151f67c640 1 --2- 172.21.15.142:0/3456641480 >> v2:172.21.15.142:6800/3892080990 conn(0x7f150403da50 0x7f150403ff10 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f15140098c0 tx=0x7f1514006db0 comp rx=0 tx=0).stop 2024-09-06T14:19:42.641 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:42.002+0000 7f151f67c640 1 -- 172.21.15.142:0/3456641480 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f151807a490 msgr2=0x7f151819a720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:19:42.641 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:42.002+0000 7f151f67c640 1 --2- 172.21.15.142:0/3456641480 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f151807a490 0x7f151819a720 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f150c006f10 tx=0x7f150c032670 comp rx=0 tx=0).stop 2024-09-06T14:19:42.641 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:42.002+0000 7f151f67c640 1 -- 172.21.15.142:0/3456641480 shutdown_connections 2024-09-06T14:19:42.642 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:42.002+0000 7f151f67c640 1 --2- 172.21.15.142:0/3456641480 >> v2:172.21.15.142:6800/3892080990 conn(0x7f150403da50 0x7f150403ff10 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:19:42.642 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:42.002+0000 7f151f67c640 1 --2- 172.21.15.142:0/3456641480 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f151807a490 0x7f151819a720 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:19:42.642 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:42.002+0000 7f151f67c640 1 -- 172.21.15.142:0/3456641480 >> 172.21.15.142:0/3456641480 conn(0x7f1518100f20 msgr2=0x7f1518101ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:19:42.642 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:42.002+0000 7f151f67c640 1 -- 172.21.15.142:0/3456641480 shutdown_connections 2024-09-06T14:19:42.642 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:42.002+0000 7f151f67c640 1 -- 172.21.15.142:0/3456641480 wait complete. 2024-09-06T14:19:42.642 INFO:teuthology.orchestra.run.smithi142.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-09-06T14:19:42.642 INFO:teuthology.orchestra.run.smithi142.stdout:Adding key to root@localhost authorized_keys... 2024-09-06T14:19:42.642 INFO:teuthology.orchestra.run.smithi142.stdout:Adding host smithi142... 2024-09-06T14:19:43.170 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:43 smithi142 ceph-mon[29958]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-06T14:19:44.456 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:44 smithi142 ceph-mon[29958]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi142", "addr": "172.21.15.142", "target": ["mon-mgr", ""]}]: dispatch 2024-09-06T14:19:44.456 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:44 smithi142 ceph-mon[29958]: Deploying cephadm binary to smithi142 2024-09-06T14:19:46.898 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout Added host 'smithi142' with addr '172.21.15.142' 2024-09-06T14:19:46.898 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:43.010+0000 7f58d9a34640 1 Processor -- start 2024-09-06T14:19:46.898 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:43.010+0000 7f58d9a34640 1 -- start start 2024-09-06T14:19:46.899 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:43.010+0000 7f58d9a34640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f58d4106710 0x7f58d4106b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:19:46.899 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:43.010+0000 7f58d9a34640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f58d4107100 con 0x7f58d4106710 2024-09-06T14:19:46.899 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:43.010+0000 7f58d2ffd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f58d4106710 0x7f58d4106b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:19:46.899 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:43.010+0000 7f58d2ffd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f58d4106710 0x7f58d4106b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:52560/0 (socket says 172.21.15.142:52560) 2024-09-06T14:19:46.899 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:43.010+0000 7f58d2ffd640 1 -- 172.21.15.142:0/3356393621 learned_addr learned my addr 172.21.15.142:0/3356393621 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:19:46.899 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:43.011+0000 7f58d2ffd640 1 -- 172.21.15.142:0/3356393621 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58d4107280 con 0x7f58d4106710 2024-09-06T14:19:46.899 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:43.011+0000 7f58d2ffd640 1 --2- 172.21.15.142:0/3356393621 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f58d4106710 0x7f58d4106b10 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f58c4009e80 tx=0x7f58c402a7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=439b685a9153c59b server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:19:46.899 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:43.012+0000 7f58d1ffb640 1 -- 172.21.15.142:0/3356393621 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f58c4036070 con 0x7f58d4106710 2024-09-06T14:19:46.899 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:43.012+0000 7f58d1ffb640 1 -- 172.21.15.142:0/3356393621 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1023+0+0 (secure 0 0 0) 0x7f58c4031860 con 0x7f58d4106710 2024-09-06T14:19:46.899 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:43.012+0000 7f58d1ffb640 1 -- 172.21.15.142:0/3356393621 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f58c4007880 con 0x7f58d4106710 2024-09-06T14:19:46.899 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:43.012+0000 7f58d9a34640 1 -- 172.21.15.142:0/3356393621 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f58d4106710 msgr2=0x7f58d4106b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:19:46.900 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:43.012+0000 7f58d9a34640 1 --2- 172.21.15.142:0/3356393621 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f58d4106710 0x7f58d4106b10 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f58c4009e80 tx=0x7f58c402a7a0 comp rx=0 tx=0).stop 2024-09-06T14:19:46.900 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:43.012+0000 7f58d9a34640 1 -- 172.21.15.142:0/3356393621 shutdown_connections 2024-09-06T14:19:46.900 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:43.012+0000 7f58d9a34640 1 --2- 172.21.15.142:0/3356393621 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f58d4106710 0x7f58d4106b10 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:19:46.900 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:43.012+0000 7f58d9a34640 1 -- 172.21.15.142:0/3356393621 >> 172.21.15.142:0/3356393621 conn(0x7f58d4102020 msgr2=0x7f58d4104440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:19:46.900 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:43.012+0000 7f58d9a34640 1 -- 172.21.15.142:0/3356393621 shutdown_connections 2024-09-06T14:19:46.900 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:43.012+0000 7f58d9a34640 1 -- 172.21.15.142:0/3356393621 wait complete. 2024-09-06T14:19:46.900 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:43.013+0000 7f58d9a34640 1 Processor -- start 2024-09-06T14:19:46.900 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:43.013+0000 7f58d9a34640 1 -- start start 2024-09-06T14:19:46.900 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:43.013+0000 7f58d9a34640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f58d4106710 0x7f58d419a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:19:46.900 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:43.013+0000 7f58d9a34640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f58d410ab40 con 0x7f58d4106710 2024-09-06T14:19:46.900 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:43.013+0000 7f58d2ffd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f58d4106710 0x7f58d419a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:19:46.901 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:43.013+0000 7f58d2ffd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f58d4106710 0x7f58d419a6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:52566/0 (socket says 172.21.15.142:52566) 2024-09-06T14:19:46.901 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:43.013+0000 7f58d2ffd640 1 -- 172.21.15.142:0/2404799418 learned_addr learned my addr 172.21.15.142:0/2404799418 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:19:46.901 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:43.014+0000 7f58d2ffd640 1 -- 172.21.15.142:0/2404799418 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58d419ac40 con 0x7f58d4106710 2024-09-06T14:19:46.901 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:43.014+0000 7f58d2ffd640 1 --2- 172.21.15.142:0/2404799418 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f58d4106710 0x7f58d419a6e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f58c4008dc0 tx=0x7f58c4008df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:19:46.901 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:43.014+0000 7f58d8a32640 1 -- 172.21.15.142:0/2404799418 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f58c4041020 con 0x7f58d4106710 2024-09-06T14:19:46.901 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:43.014+0000 7f58d8a32640 1 -- 172.21.15.142:0/2404799418 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1023+0+0 (secure 0 0 0) 0x7f58c4031a10 con 0x7f58d4106710 2024-09-06T14:19:46.901 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:43.014+0000 7f58d9a34640 1 -- 172.21.15.142:0/2404799418 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58d419aed0 con 0x7f58d4106710 2024-09-06T14:19:46.901 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:43.014+0000 7f58d8a32640 1 -- 172.21.15.142:0/2404799418 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f58c4036040 con 0x7f58d4106710 2024-09-06T14:19:46.902 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:43.014+0000 7f58d9a34640 1 -- 172.21.15.142:0/2404799418 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58d419b910 con 0x7f58d4106710 2024-09-06T14:19:46.902 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:43.015+0000 7f58d8a32640 1 -- 172.21.15.142:0/2404799418 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 8) ==== 50075+0+0 (secure 0 0 0) 0x7f58c4044020 con 0x7f58d4106710 2024-09-06T14:19:46.902 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:43.015+0000 7f58d9a34640 1 -- 172.21.15.142:0/2404799418 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58d4106b90 con 0x7f58d4106710 2024-09-06T14:19:46.902 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:43.015+0000 7f58d8a32640 1 --2- 172.21.15.142:0/2404799418 >> v2:172.21.15.142:6800/3892080990 conn(0x7f58bc03da00 0x7f58bc03fec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:19:46.902 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:43.015+0000 7f58d8a32640 1 -- 172.21.15.142:0/2404799418 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 854+0+0 (secure 0 0 0) 0x7f58c4072fe0 con 0x7f58d4106710 2024-09-06T14:19:46.902 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:43.015+0000 7f58d27fc640 1 --2- 172.21.15.142:0/2404799418 >> v2:172.21.15.142:6800/3892080990 conn(0x7f58bc03da00 0x7f58bc03fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:19:46.903 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:43.016+0000 7f58d27fc640 1 --2- 172.21.15.142:0/2404799418 >> v2:172.21.15.142:6800/3892080990 conn(0x7f58bc03da00 0x7f58bc03fec0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f58c80098c0 tx=0x7f58c8006db0 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:19:46.903 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:43.019+0000 7f58d8a32640 1 -- 172.21.15.142:0/2404799418 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7f58c404b020 con 0x7f58d4106710 2024-09-06T14:19:46.903 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:43.175+0000 7f58d9a34640 1 -- 172.21.15.142:0/2404799418 --> v2:172.21.15.142:6800/3892080990 -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi142", "addr": "172.21.15.142", "target": ["mon-mgr", ""]}) -- 0x7f58d419c1b0 con 0x7f58bc03da00 2024-09-06T14:19:46.903 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:46.325+0000 7f58d8a32640 1 -- 172.21.15.142:0/2404799418 <== mgr.14116 v2:172.21.15.142:6800/3892080990 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f58d419c1b0 con 0x7f58bc03da00 2024-09-06T14:19:46.903 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:46.328+0000 7f58d9a34640 1 -- 172.21.15.142:0/2404799418 >> v2:172.21.15.142:6800/3892080990 conn(0x7f58bc03da00 msgr2=0x7f58bc03fec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:19:46.903 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:46.328+0000 7f58d9a34640 1 --2- 172.21.15.142:0/2404799418 >> v2:172.21.15.142:6800/3892080990 conn(0x7f58bc03da00 0x7f58bc03fec0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f58c80098c0 tx=0x7f58c8006db0 comp rx=0 tx=0).stop 2024-09-06T14:19:46.903 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:46.328+0000 7f58d9a34640 1 -- 172.21.15.142:0/2404799418 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f58d4106710 msgr2=0x7f58d419a6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:19:46.903 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:46.328+0000 7f58d9a34640 1 --2- 172.21.15.142:0/2404799418 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f58d4106710 0x7f58d419a6e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f58c4008dc0 tx=0x7f58c4008df0 comp rx=0 tx=0).stop 2024-09-06T14:19:46.903 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:46.328+0000 7f58d9a34640 1 -- 172.21.15.142:0/2404799418 shutdown_connections 2024-09-06T14:19:46.903 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:46.328+0000 7f58d9a34640 1 --2- 172.21.15.142:0/2404799418 >> v2:172.21.15.142:6800/3892080990 conn(0x7f58bc03da00 0x7f58bc03fec0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:19:46.903 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:46.328+0000 7f58d9a34640 1 --2- 172.21.15.142:0/2404799418 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f58d4106710 0x7f58d419a6e0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:19:46.903 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:46.328+0000 7f58d9a34640 1 -- 172.21.15.142:0/2404799418 >> 172.21.15.142:0/2404799418 conn(0x7f58d4102020 msgr2=0x7f58d4102b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:19:46.903 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:46.328+0000 7f58d9a34640 1 -- 172.21.15.142:0/2404799418 shutdown_connections 2024-09-06T14:19:46.903 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:46.328+0000 7f58d9a34640 1 -- 172.21.15.142:0/2404799418 wait complete. 2024-09-06T14:19:46.904 INFO:teuthology.orchestra.run.smithi142.stdout:Deploying unmanaged mon service... 2024-09-06T14:19:47.508 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:47 smithi142 ceph-mon[29958]: from='mgr.14116 172.21.15.142:0/3534094648' entity='mgr.y' 2024-09-06T14:19:47.508 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:47 smithi142 ceph-mon[29958]: Added host smithi142 2024-09-06T14:19:47.508 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:47 smithi142 ceph-mon[29958]: from='mgr.14116 172.21.15.142:0/3534094648' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-06T14:19:48.315 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-09-06T14:19:48.315 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:47.319+0000 7f0b48b55640 1 Processor -- start 2024-09-06T14:19:48.315 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:47.319+0000 7f0b48b55640 1 -- start start 2024-09-06T14:19:48.315 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:47.319+0000 7f0b48b55640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0b441055b0 0x7f0b441059b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:19:48.315 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:47.319+0000 7f0b48b55640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f0b44105fa0 con 0x7f0b441055b0 2024-09-06T14:19:48.316 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:47.319+0000 7f0b4259c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0b441055b0 0x7f0b441059b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:19:48.316 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:47.319+0000 7f0b4259c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0b441055b0 0x7f0b441059b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:34998/0 (socket says 172.21.15.142:34998) 2024-09-06T14:19:48.316 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:47.319+0000 7f0b4259c640 1 -- 172.21.15.142:0/3468737501 learned_addr learned my addr 172.21.15.142:0/3468737501 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:19:48.316 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:47.320+0000 7f0b4259c640 1 -- 172.21.15.142:0/3468737501 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b44106120 con 0x7f0b441055b0 2024-09-06T14:19:48.316 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:47.320+0000 7f0b4259c640 1 --2- 172.21.15.142:0/3468737501 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0b441055b0 0x7f0b441059b0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f0b34009820 tx=0x7f0b3402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d8249ed737ea98b server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:19:48.316 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:47.320+0000 7f0b4159a640 1 -- 172.21.15.142:0/3468737501 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0b34037070 con 0x7f0b441055b0 2024-09-06T14:19:48.316 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:47.321+0000 7f0b4159a640 1 -- 172.21.15.142:0/3468737501 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1023+0+0 (secure 0 0 0) 0x7f0b340328a0 con 0x7f0b441055b0 2024-09-06T14:19:48.316 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:47.321+0000 7f0b4159a640 1 -- 172.21.15.142:0/3468737501 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0b34030690 con 0x7f0b441055b0 2024-09-06T14:19:48.316 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:47.321+0000 7f0b48b55640 1 -- 172.21.15.142:0/3468737501 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0b441055b0 msgr2=0x7f0b441059b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:19:48.316 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:47.321+0000 7f0b48b55640 1 --2- 172.21.15.142:0/3468737501 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0b441055b0 0x7f0b441059b0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f0b34009820 tx=0x7f0b3402a270 comp rx=0 tx=0).stop 2024-09-06T14:19:48.316 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:47.321+0000 7f0b48b55640 1 -- 172.21.15.142:0/3468737501 shutdown_connections 2024-09-06T14:19:48.316 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:47.321+0000 7f0b48b55640 1 --2- 172.21.15.142:0/3468737501 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0b441055b0 0x7f0b441059b0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:19:48.316 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:47.321+0000 7f0b48b55640 1 -- 172.21.15.142:0/3468737501 >> 172.21.15.142:0/3468737501 conn(0x7f0b44100ea0 msgr2=0x7f0b441032c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:19:48.317 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:47.321+0000 7f0b48b55640 1 -- 172.21.15.142:0/3468737501 shutdown_connections 2024-09-06T14:19:48.317 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:47.321+0000 7f0b48b55640 1 -- 172.21.15.142:0/3468737501 wait complete. 2024-09-06T14:19:48.317 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:47.322+0000 7f0b48b55640 1 Processor -- start 2024-09-06T14:19:48.317 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:47.322+0000 7f0b48b55640 1 -- start start 2024-09-06T14:19:48.317 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:47.322+0000 7f0b48b55640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0b441055b0 0x7f0b4419a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:19:48.317 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:47.322+0000 7f0b48b55640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f0b44109a30 con 0x7f0b441055b0 2024-09-06T14:19:48.317 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:47.322+0000 7f0b4259c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0b441055b0 0x7f0b4419a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:19:48.317 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:47.322+0000 7f0b4259c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0b441055b0 0x7f0b4419a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:35004/0 (socket says 172.21.15.142:35004) 2024-09-06T14:19:48.317 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:47.322+0000 7f0b4259c640 1 -- 172.21.15.142:0/1114883639 learned_addr learned my addr 172.21.15.142:0/1114883639 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:19:48.317 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:47.323+0000 7f0b4259c640 1 -- 172.21.15.142:0/1114883639 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b4419ac70 con 0x7f0b441055b0 2024-09-06T14:19:48.317 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:47.323+0000 7f0b4259c640 1 --2- 172.21.15.142:0/1114883639 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0b441055b0 0x7f0b4419a710 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f0b34000c00 tx=0x7f0b34032740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:19:48.317 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:47.323+0000 7f0b337fe640 1 -- 172.21.15.142:0/1114883639 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0b34037040 con 0x7f0b441055b0 2024-09-06T14:19:48.317 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:47.323+0000 7f0b337fe640 1 -- 172.21.15.142:0/1114883639 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1023+0+0 (secure 0 0 0) 0x7f0b34040070 con 0x7f0b441055b0 2024-09-06T14:19:48.318 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:47.323+0000 7f0b48b55640 1 -- 172.21.15.142:0/1114883639 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b4419af00 con 0x7f0b441055b0 2024-09-06T14:19:48.318 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:47.323+0000 7f0b337fe640 1 -- 172.21.15.142:0/1114883639 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0b3403be40 con 0x7f0b441055b0 2024-09-06T14:19:48.318 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:47.323+0000 7f0b48b55640 1 -- 172.21.15.142:0/1114883639 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b4419b8c0 con 0x7f0b441055b0 2024-09-06T14:19:48.318 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:47.324+0000 7f0b337fe640 1 -- 172.21.15.142:0/1114883639 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 8) ==== 50075+0+0 (secure 0 0 0) 0x7f0b34045460 con 0x7f0b441055b0 2024-09-06T14:19:48.318 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:47.324+0000 7f0b48b55640 1 -- 172.21.15.142:0/1114883639 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b44105a30 con 0x7f0b441055b0 2024-09-06T14:19:48.318 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:47.324+0000 7f0b337fe640 1 --2- 172.21.15.142:0/1114883639 >> v2:172.21.15.142:6800/3892080990 conn(0x7f0b2c03d9e0 0x7f0b2c03fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:19:48.318 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:47.324+0000 7f0b337fe640 1 -- 172.21.15.142:0/1114883639 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 854+0+0 (secure 0 0 0) 0x7f0b34071a10 con 0x7f0b441055b0 2024-09-06T14:19:48.318 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:47.324+0000 7f0b41d9b640 1 --2- 172.21.15.142:0/1114883639 >> v2:172.21.15.142:6800/3892080990 conn(0x7f0b2c03d9e0 0x7f0b2c03fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:19:48.318 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:47.325+0000 7f0b41d9b640 1 --2- 172.21.15.142:0/1114883639 >> v2:172.21.15.142:6800/3892080990 conn(0x7f0b2c03d9e0 0x7f0b2c03fea0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f0b380098c0 tx=0x7f0b38006db0 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:19:48.318 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:47.331+0000 7f0b337fe640 1 -- 172.21.15.142:0/1114883639 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7f0b3403b430 con 0x7f0b441055b0 2024-09-06T14:19:48.319 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:47.512+0000 7f0b48b55640 1 -- 172.21.15.142:0/1114883639 --> v2:172.21.15.142:6800/3892080990 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f0b4419c110 con 0x7f0b2c03d9e0 2024-09-06T14:19:48.319 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:47.519+0000 7f0b337fe640 1 -- 172.21.15.142:0/1114883639 <== mgr.14116 v2:172.21.15.142:6800/3892080990 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f0b4419c110 con 0x7f0b2c03d9e0 2024-09-06T14:19:48.319 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:47.522+0000 7f0b48b55640 1 -- 172.21.15.142:0/1114883639 >> v2:172.21.15.142:6800/3892080990 conn(0x7f0b2c03d9e0 msgr2=0x7f0b2c03fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:19:48.319 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:47.522+0000 7f0b48b55640 1 --2- 172.21.15.142:0/1114883639 >> v2:172.21.15.142:6800/3892080990 conn(0x7f0b2c03d9e0 0x7f0b2c03fea0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f0b380098c0 tx=0x7f0b38006db0 comp rx=0 tx=0).stop 2024-09-06T14:19:48.319 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:47.522+0000 7f0b48b55640 1 -- 172.21.15.142:0/1114883639 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0b441055b0 msgr2=0x7f0b4419a710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:19:48.319 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:47.522+0000 7f0b48b55640 1 --2- 172.21.15.142:0/1114883639 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0b441055b0 0x7f0b4419a710 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f0b34000c00 tx=0x7f0b34032740 comp rx=0 tx=0).stop 2024-09-06T14:19:48.319 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:47.522+0000 7f0b48b55640 1 -- 172.21.15.142:0/1114883639 shutdown_connections 2024-09-06T14:19:48.319 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:47.522+0000 7f0b48b55640 1 --2- 172.21.15.142:0/1114883639 >> v2:172.21.15.142:6800/3892080990 conn(0x7f0b2c03d9e0 0x7f0b2c03fea0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:19:48.319 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:47.522+0000 7f0b48b55640 1 --2- 172.21.15.142:0/1114883639 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0b441055b0 0x7f0b4419a710 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:19:48.319 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:47.522+0000 7f0b48b55640 1 -- 172.21.15.142:0/1114883639 >> 172.21.15.142:0/1114883639 conn(0x7f0b44100ea0 msgr2=0x7f0b44079930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:19:48.319 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:47.522+0000 7f0b48b55640 1 -- 172.21.15.142:0/1114883639 shutdown_connections 2024-09-06T14:19:48.319 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:47.522+0000 7f0b48b55640 1 -- 172.21.15.142:0/1114883639 wait complete. 2024-09-06T14:19:48.319 INFO:teuthology.orchestra.run.smithi142.stdout:Deploying unmanaged mgr service... 2024-09-06T14:19:48.877 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:48 smithi142 ceph-mon[29958]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-06T14:19:48.877 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:48 smithi142 ceph-mon[29958]: Saving service mon spec with placement count:5 2024-09-06T14:19:48.877 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:48 smithi142 ceph-mon[29958]: from='mgr.14116 172.21.15.142:0/3534094648' entity='mgr.y' 2024-09-06T14:19:49.606 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-09-06T14:19:49.606 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:48.738+0000 7fc0f35ac640 1 Processor -- start 2024-09-06T14:19:49.606 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:48.738+0000 7fc0f35ac640 1 -- start start 2024-09-06T14:19:49.607 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:48.738+0000 7fc0f35ac640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc0ec07a490 0x7fc0ec1092f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:19:49.607 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:48.738+0000 7fc0f35ac640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fc0ec109830 con 0x7fc0ec07a490 2024-09-06T14:19:49.607 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:48.738+0000 7fc0f1348640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc0ec07a490 0x7fc0ec1092f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:19:49.607 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:48.738+0000 7fc0f1348640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc0ec07a490 0x7fc0ec1092f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:35018/0 (socket says 172.21.15.142:35018) 2024-09-06T14:19:49.607 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:48.738+0000 7fc0f1348640 1 -- 172.21.15.142:0/2331609373 learned_addr learned my addr 172.21.15.142:0/2331609373 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:19:49.607 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:48.738+0000 7fc0f1348640 1 -- 172.21.15.142:0/2331609373 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0ec1099b0 con 0x7fc0ec07a490 2024-09-06T14:19:49.607 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:48.739+0000 7fc0f1348640 1 --2- 172.21.15.142:0/2331609373 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc0ec07a490 0x7fc0ec1092f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc0e0009820 tx=0x7fc0e002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1e3ec5d8ddc865 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:19:49.607 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:48.739+0000 7fc0dffff640 1 -- 172.21.15.142:0/2331609373 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc0e0037070 con 0x7fc0ec07a490 2024-09-06T14:19:49.607 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:48.739+0000 7fc0dffff640 1 -- 172.21.15.142:0/2331609373 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1023+0+0 (secure 0 0 0) 0x7fc0e00328a0 con 0x7fc0ec07a490 2024-09-06T14:19:49.607 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:48.739+0000 7fc0dffff640 1 -- 172.21.15.142:0/2331609373 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc0e0030690 con 0x7fc0ec07a490 2024-09-06T14:19:49.607 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:48.740+0000 7fc0f35ac640 1 -- 172.21.15.142:0/2331609373 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc0ec07a490 msgr2=0x7fc0ec1092f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:19:49.607 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:48.740+0000 7fc0f35ac640 1 --2- 172.21.15.142:0/2331609373 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc0ec07a490 0x7fc0ec1092f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc0e0009820 tx=0x7fc0e002a270 comp rx=0 tx=0).stop 2024-09-06T14:19:49.607 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:48.740+0000 7fc0f35ac640 1 -- 172.21.15.142:0/2331609373 shutdown_connections 2024-09-06T14:19:49.608 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:48.740+0000 7fc0f35ac640 1 --2- 172.21.15.142:0/2331609373 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc0ec07a490 0x7fc0ec1092f0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:19:49.608 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:48.740+0000 7fc0f35ac640 1 -- 172.21.15.142:0/2331609373 >> 172.21.15.142:0/2331609373 conn(0x7fc0ec100ee0 msgr2=0x7fc0ec103300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:19:49.608 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:48.740+0000 7fc0f35ac640 1 -- 172.21.15.142:0/2331609373 shutdown_connections 2024-09-06T14:19:49.608 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:48.740+0000 7fc0f35ac640 1 -- 172.21.15.142:0/2331609373 wait complete. 2024-09-06T14:19:49.608 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:48.740+0000 7fc0f35ac640 1 Processor -- start 2024-09-06T14:19:49.608 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:48.740+0000 7fc0f35ac640 1 -- start start 2024-09-06T14:19:49.608 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:48.740+0000 7fc0f35ac640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc0ec07a490 0x7fc0ec19a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:19:49.608 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:48.740+0000 7fc0f35ac640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fc0ec10a9b0 con 0x7fc0ec07a490 2024-09-06T14:19:49.608 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:48.741+0000 7fc0f1348640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc0ec07a490 0x7fc0ec19a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:19:49.608 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:48.741+0000 7fc0f1348640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc0ec07a490 0x7fc0ec19a6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:35026/0 (socket says 172.21.15.142:35026) 2024-09-06T14:19:49.608 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:48.741+0000 7fc0f1348640 1 -- 172.21.15.142:0/388536913 learned_addr learned my addr 172.21.15.142:0/388536913 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:19:49.608 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:48.741+0000 7fc0f1348640 1 -- 172.21.15.142:0/388536913 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0ec19ac00 con 0x7fc0ec07a490 2024-09-06T14:19:49.608 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:48.741+0000 7fc0f1348640 1 --2- 172.21.15.142:0/388536913 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc0ec07a490 0x7fc0ec19a6a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fc0e0006f10 tx=0x7fc0e0032670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:19:49.609 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:48.741+0000 7fc0de7fc640 1 -- 172.21.15.142:0/388536913 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc0e0037040 con 0x7fc0ec07a490 2024-09-06T14:19:49.609 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:48.741+0000 7fc0de7fc640 1 -- 172.21.15.142:0/388536913 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1023+0+0 (secure 0 0 0) 0x7fc0e0040070 con 0x7fc0ec07a490 2024-09-06T14:19:49.609 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:48.741+0000 7fc0f35ac640 1 -- 172.21.15.142:0/388536913 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0ec19ae90 con 0x7fc0ec07a490 2024-09-06T14:19:49.609 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:48.741+0000 7fc0de7fc640 1 -- 172.21.15.142:0/388536913 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc0e003a390 con 0x7fc0ec07a490 2024-09-06T14:19:49.609 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:48.741+0000 7fc0f35ac640 1 -- 172.21.15.142:0/388536913 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0ec19b850 con 0x7fc0ec07a490 2024-09-06T14:19:49.609 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:48.742+0000 7fc0de7fc640 1 -- 172.21.15.142:0/388536913 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 8) ==== 50075+0+0 (secure 0 0 0) 0x7fc0e003a600 con 0x7fc0ec07a490 2024-09-06T14:19:49.609 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:48.742+0000 7fc0f35ac640 1 -- 172.21.15.142:0/388536913 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0ec107360 con 0x7fc0ec07a490 2024-09-06T14:19:49.609 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:48.742+0000 7fc0de7fc640 1 --2- 172.21.15.142:0/388536913 >> v2:172.21.15.142:6800/3892080990 conn(0x7fc0d803da50 0x7fc0d803ff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:19:49.609 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:48.742+0000 7fc0de7fc640 1 -- 172.21.15.142:0/388536913 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 854+0+0 (secure 0 0 0) 0x7fc0e0071b20 con 0x7fc0ec07a490 2024-09-06T14:19:49.609 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:48.742+0000 7fc0f0b47640 1 --2- 172.21.15.142:0/388536913 >> v2:172.21.15.142:6800/3892080990 conn(0x7fc0d803da50 0x7fc0d803ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:19:49.609 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:48.743+0000 7fc0f0b47640 1 --2- 172.21.15.142:0/388536913 >> v2:172.21.15.142:6800/3892080990 conn(0x7fc0d803da50 0x7fc0d803ff10 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc0e80098c0 tx=0x7fc0e8006db0 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:19:49.609 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:48.746+0000 7fc0de7fc640 1 -- 172.21.15.142:0/388536913 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7fc0e0042ce0 con 0x7fc0ec07a490 2024-09-06T14:19:49.609 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:48.915+0000 7fc0f35ac640 1 -- 172.21.15.142:0/388536913 --> v2:172.21.15.142:6800/3892080990 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7fc0ec102150 con 0x7fc0d803da50 2024-09-06T14:19:49.610 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:48.922+0000 7fc0de7fc640 1 -- 172.21.15.142:0/388536913 <== mgr.14116 v2:172.21.15.142:6800/3892080990 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fc0ec102150 con 0x7fc0d803da50 2024-09-06T14:19:49.610 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:48.924+0000 7fc0f35ac640 1 -- 172.21.15.142:0/388536913 >> v2:172.21.15.142:6800/3892080990 conn(0x7fc0d803da50 msgr2=0x7fc0d803ff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:19:49.610 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:48.924+0000 7fc0f35ac640 1 --2- 172.21.15.142:0/388536913 >> v2:172.21.15.142:6800/3892080990 conn(0x7fc0d803da50 0x7fc0d803ff10 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc0e80098c0 tx=0x7fc0e8006db0 comp rx=0 tx=0).stop 2024-09-06T14:19:49.610 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:48.924+0000 7fc0f35ac640 1 -- 172.21.15.142:0/388536913 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc0ec07a490 msgr2=0x7fc0ec19a6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:19:49.610 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:48.924+0000 7fc0f35ac640 1 --2- 172.21.15.142:0/388536913 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc0ec07a490 0x7fc0ec19a6a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fc0e0006f10 tx=0x7fc0e0032670 comp rx=0 tx=0).stop 2024-09-06T14:19:49.610 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:48.924+0000 7fc0f35ac640 1 -- 172.21.15.142:0/388536913 shutdown_connections 2024-09-06T14:19:49.610 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:48.924+0000 7fc0f35ac640 1 --2- 172.21.15.142:0/388536913 >> v2:172.21.15.142:6800/3892080990 conn(0x7fc0d803da50 0x7fc0d803ff10 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:19:49.610 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:48.924+0000 7fc0f35ac640 1 --2- 172.21.15.142:0/388536913 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc0ec07a490 0x7fc0ec19a6a0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:19:49.610 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:48.924+0000 7fc0f35ac640 1 -- 172.21.15.142:0/388536913 >> 172.21.15.142:0/388536913 conn(0x7fc0ec100ee0 msgr2=0x7fc0ec101a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:19:49.610 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:48.924+0000 7fc0f35ac640 1 -- 172.21.15.142:0/388536913 shutdown_connections 2024-09-06T14:19:49.610 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:48.924+0000 7fc0f35ac640 1 -- 172.21.15.142:0/388536913 wait complete. 2024-09-06T14:19:50.159 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:49 smithi142 ceph-mon[29958]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-06T14:19:50.160 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:49 smithi142 ceph-mon[29958]: Saving service mgr spec with placement count:2 2024-09-06T14:19:50.160 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:49 smithi142 ceph-mon[29958]: from='mgr.14116 172.21.15.142:0/3534094648' entity='mgr.y' 2024-09-06T14:19:51.291 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:51 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/3699077227' entity='client.admin' 2024-09-06T14:19:51.292 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:51 smithi142 ceph-mon[29958]: from='mgr.14116 172.21.15.142:0/3534094648' entity='mgr.y' 2024-09-06T14:19:51.299 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:50.003+0000 7fe40cc8c640 1 Processor -- start 2024-09-06T14:19:51.300 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:50.003+0000 7fe40cc8c640 1 -- start start 2024-09-06T14:19:51.300 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:50.003+0000 7fe40cc8c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe408106710 0x7fe408106b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:19:51.300 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:50.003+0000 7fe40cc8c640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fe408107100 con 0x7fe408106710 2024-09-06T14:19:51.300 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:50.003+0000 7fe40659c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe408106710 0x7fe408106b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:19:51.300 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:50.003+0000 7fe40659c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe408106710 0x7fe408106b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:35034/0 (socket says 172.21.15.142:35034) 2024-09-06T14:19:51.301 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:50.003+0000 7fe40659c640 1 -- 172.21.15.142:0/3527459192 learned_addr learned my addr 172.21.15.142:0/3527459192 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:19:51.301 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:50.004+0000 7fe40659c640 1 -- 172.21.15.142:0/3527459192 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe408107280 con 0x7fe408106710 2024-09-06T14:19:51.301 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:50.004+0000 7fe40659c640 1 --2- 172.21.15.142:0/3527459192 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe408106710 0x7fe408106b10 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe3f8009820 tx=0x7fe3f802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fa3789b4bf9d826 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:19:51.301 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:50.004+0000 7fe40559a640 1 -- 172.21.15.142:0/3527459192 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe3f8037070 con 0x7fe408106710 2024-09-06T14:19:51.301 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:50.004+0000 7fe40559a640 1 -- 172.21.15.142:0/3527459192 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1023+0+0 (secure 0 0 0) 0x7fe3f80328a0 con 0x7fe408106710 2024-09-06T14:19:51.301 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:50.004+0000 7fe40559a640 1 -- 172.21.15.142:0/3527459192 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe3f8030690 con 0x7fe408106710 2024-09-06T14:19:51.302 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:50.005+0000 7fe40cc8c640 1 -- 172.21.15.142:0/3527459192 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe408106710 msgr2=0x7fe408106b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:19:51.302 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:50.005+0000 7fe40cc8c640 1 --2- 172.21.15.142:0/3527459192 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe408106710 0x7fe408106b10 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe3f8009820 tx=0x7fe3f802a270 comp rx=0 tx=0).stop 2024-09-06T14:19:51.302 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:50.005+0000 7fe40cc8c640 1 -- 172.21.15.142:0/3527459192 shutdown_connections 2024-09-06T14:19:51.302 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:50.005+0000 7fe40cc8c640 1 --2- 172.21.15.142:0/3527459192 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe408106710 0x7fe408106b10 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:19:51.302 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:50.005+0000 7fe40cc8c640 1 -- 172.21.15.142:0/3527459192 >> 172.21.15.142:0/3527459192 conn(0x7fe408102020 msgr2=0x7fe408104440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:19:51.303 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:50.005+0000 7fe40cc8c640 1 -- 172.21.15.142:0/3527459192 shutdown_connections 2024-09-06T14:19:51.303 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:50.005+0000 7fe40cc8c640 1 -- 172.21.15.142:0/3527459192 wait complete. 2024-09-06T14:19:51.303 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:50.006+0000 7fe40cc8c640 1 Processor -- start 2024-09-06T14:19:51.303 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:50.006+0000 7fe40cc8c640 1 -- start start 2024-09-06T14:19:51.303 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:50.006+0000 7fe40cc8c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe408106710 0x7fe40819a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:19:51.303 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:50.006+0000 7fe40cc8c640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fe40810ab40 con 0x7fe408106710 2024-09-06T14:19:51.304 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:50.006+0000 7fe40659c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe408106710 0x7fe40819a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:19:51.304 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:50.006+0000 7fe40659c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe408106710 0x7fe40819a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:35044/0 (socket says 172.21.15.142:35044) 2024-09-06T14:19:51.304 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:50.006+0000 7fe40659c640 1 -- 172.21.15.142:0/3699077227 learned_addr learned my addr 172.21.15.142:0/3699077227 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:19:51.304 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:50.006+0000 7fe40659c640 1 -- 172.21.15.142:0/3699077227 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe40819ac70 con 0x7fe408106710 2024-09-06T14:19:51.304 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:50.007+0000 7fe40659c640 1 --2- 172.21.15.142:0/3699077227 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe408106710 0x7fe40819a710 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe3f8000c00 tx=0x7fe3f8032740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:19:51.304 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:50.007+0000 7fe3f77fe640 1 -- 172.21.15.142:0/3699077227 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe3f8037040 con 0x7fe408106710 2024-09-06T14:19:51.305 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:50.007+0000 7fe3f77fe640 1 -- 172.21.15.142:0/3699077227 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1023+0+0 (secure 0 0 0) 0x7fe3f8040070 con 0x7fe408106710 2024-09-06T14:19:51.305 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:50.007+0000 7fe40cc8c640 1 -- 172.21.15.142:0/3699077227 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe40819af00 con 0x7fe408106710 2024-09-06T14:19:51.305 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:50.007+0000 7fe3f77fe640 1 -- 172.21.15.142:0/3699077227 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe3f803be40 con 0x7fe408106710 2024-09-06T14:19:51.305 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:50.007+0000 7fe40cc8c640 1 -- 172.21.15.142:0/3699077227 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe40819b870 con 0x7fe408106710 2024-09-06T14:19:51.305 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:50.008+0000 7fe3f77fe640 1 -- 172.21.15.142:0/3699077227 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 8) ==== 50075+0+0 (secure 0 0 0) 0x7fe3f8045460 con 0x7fe408106710 2024-09-06T14:19:51.305 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:50.008+0000 7fe40cc8c640 1 -- 172.21.15.142:0/3699077227 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe408106b90 con 0x7fe408106710 2024-09-06T14:19:51.306 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:50.008+0000 7fe3f77fe640 1 --2- 172.21.15.142:0/3699077227 >> v2:172.21.15.142:6800/3892080990 conn(0x7fe3f003da50 0x7fe3f003ff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:19:51.306 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:50.008+0000 7fe3f77fe640 1 -- 172.21.15.142:0/3699077227 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 854+0+0 (secure 0 0 0) 0x7fe3f80719c0 con 0x7fe408106710 2024-09-06T14:19:51.306 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:50.008+0000 7fe405d9b640 1 --2- 172.21.15.142:0/3699077227 >> v2:172.21.15.142:6800/3892080990 conn(0x7fe3f003da50 0x7fe3f003ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:19:51.306 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:50.009+0000 7fe405d9b640 1 --2- 172.21.15.142:0/3699077227 >> v2:172.21.15.142:6800/3892080990 conn(0x7fe3f003da50 0x7fe3f003ff10 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fe3fc0098c0 tx=0x7fe3fc006db0 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:19:51.306 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:50.012+0000 7fe3f77fe640 1 -- 172.21.15.142:0/3699077227 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7fe3f803b430 con 0x7fe408106710 2024-09-06T14:19:51.306 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:50.164+0000 7fe40cc8c640 1 -- 172.21.15.142:0/3699077227 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7fe40819c050 con 0x7fe408106710 2024-09-06T14:19:51.307 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:50.169+0000 7fe3f77fe640 1 -- 172.21.15.142:0/3699077227 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v6)=0 v6) ==== 142+0+0 (secure 0 0 0) 0x7fe3f8045750 con 0x7fe408106710 2024-09-06T14:19:51.307 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:50.172+0000 7fe40cc8c640 1 -- 172.21.15.142:0/3699077227 >> v2:172.21.15.142:6800/3892080990 conn(0x7fe3f003da50 msgr2=0x7fe3f003ff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:19:51.307 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:50.172+0000 7fe40cc8c640 1 --2- 172.21.15.142:0/3699077227 >> v2:172.21.15.142:6800/3892080990 conn(0x7fe3f003da50 0x7fe3f003ff10 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fe3fc0098c0 tx=0x7fe3fc006db0 comp rx=0 tx=0).stop 2024-09-06T14:19:51.307 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:50.172+0000 7fe40cc8c640 1 -- 172.21.15.142:0/3699077227 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe408106710 msgr2=0x7fe40819a710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:19:51.307 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:50.172+0000 7fe40cc8c640 1 --2- 172.21.15.142:0/3699077227 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe408106710 0x7fe40819a710 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe3f8000c00 tx=0x7fe3f8032740 comp rx=0 tx=0).stop 2024-09-06T14:19:51.308 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:50.172+0000 7fe40cc8c640 1 -- 172.21.15.142:0/3699077227 shutdown_connections 2024-09-06T14:19:51.308 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:50.172+0000 7fe40cc8c640 1 --2- 172.21.15.142:0/3699077227 >> v2:172.21.15.142:6800/3892080990 conn(0x7fe3f003da50 0x7fe3f003ff10 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:19:51.308 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:50.172+0000 7fe40cc8c640 1 --2- 172.21.15.142:0/3699077227 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe408106710 0x7fe40819a710 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:19:51.308 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:50.172+0000 7fe40cc8c640 1 -- 172.21.15.142:0/3699077227 >> 172.21.15.142:0/3699077227 conn(0x7fe408102020 msgr2=0x7fe408102b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:19:51.308 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:50.172+0000 7fe40cc8c640 1 -- 172.21.15.142:0/3699077227 shutdown_connections 2024-09-06T14:19:51.309 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:50.172+0000 7fe40cc8c640 1 -- 172.21.15.142:0/3699077227 wait complete. 2024-09-06T14:19:52.480 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:51.695+0000 7fb9bcf95640 1 Processor -- start 2024-09-06T14:19:52.480 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:51.696+0000 7fb9bcf95640 1 -- start start 2024-09-06T14:19:52.480 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:51.696+0000 7fb9bcf95640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb9b8104a70 0x7fb9b8104e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:19:52.480 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:51.696+0000 7fb9bcf95640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fb9b8105440 con 0x7fb9b8104a70 2024-09-06T14:19:52.480 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:51.696+0000 7fb9b659c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb9b8104a70 0x7fb9b8104e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:19:52.480 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:51.696+0000 7fb9b659c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb9b8104a70 0x7fb9b8104e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:35056/0 (socket says 172.21.15.142:35056) 2024-09-06T14:19:52.481 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:51.696+0000 7fb9b659c640 1 -- 172.21.15.142:0/3863437779 learned_addr learned my addr 172.21.15.142:0/3863437779 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:19:52.481 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:51.696+0000 7fb9b659c640 1 -- 172.21.15.142:0/3863437779 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9b81055c0 con 0x7fb9b8104a70 2024-09-06T14:19:52.481 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:51.697+0000 7fb9b659c640 1 --2- 172.21.15.142:0/3863437779 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb9b8104a70 0x7fb9b8104e50 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fb9a8009820 tx=0x7fb9a802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e7fac4d061592f8 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:19:52.481 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:51.697+0000 7fb9b5d9b640 1 -- 172.21.15.142:0/3863437779 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb9a8037070 con 0x7fb9b8104a70 2024-09-06T14:19:52.481 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:51.697+0000 7fb9b5d9b640 1 -- 172.21.15.142:0/3863437779 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7fb9a80328c0 con 0x7fb9b8104a70 2024-09-06T14:19:52.481 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:51.697+0000 7fb9b5d9b640 1 -- 172.21.15.142:0/3863437779 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb9a8030640 con 0x7fb9b8104a70 2024-09-06T14:19:52.481 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:51.698+0000 7fb9bcf95640 1 -- 172.21.15.142:0/3863437779 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb9b8104a70 msgr2=0x7fb9b8104e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:19:52.481 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:51.698+0000 7fb9bcf95640 1 --2- 172.21.15.142:0/3863437779 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb9b8104a70 0x7fb9b8104e50 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fb9a8009820 tx=0x7fb9a802a270 comp rx=0 tx=0).stop 2024-09-06T14:19:52.481 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:51.698+0000 7fb9bcf95640 1 -- 172.21.15.142:0/3863437779 shutdown_connections 2024-09-06T14:19:52.481 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:51.698+0000 7fb9bcf95640 1 --2- 172.21.15.142:0/3863437779 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb9b8104a70 0x7fb9b8104e50 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:19:52.481 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:51.698+0000 7fb9bcf95640 1 -- 172.21.15.142:0/3863437779 >> 172.21.15.142:0/3863437779 conn(0x7fb9b8100700 msgr2=0x7fb9b8102b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:19:52.481 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:51.698+0000 7fb9bcf95640 1 -- 172.21.15.142:0/3863437779 shutdown_connections 2024-09-06T14:19:52.481 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:51.698+0000 7fb9bcf95640 1 -- 172.21.15.142:0/3863437779 wait complete. 2024-09-06T14:19:52.481 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:51.698+0000 7fb9bcf95640 1 Processor -- start 2024-09-06T14:19:52.482 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:51.698+0000 7fb9bcf95640 1 -- start start 2024-09-06T14:19:52.482 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:51.699+0000 7fb9bcf95640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb9b8104a70 0x7fb9b819a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:19:52.482 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:51.699+0000 7fb9bcf95640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fb9b810aa50 con 0x7fb9b8104a70 2024-09-06T14:19:52.482 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:51.699+0000 7fb9b659c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb9b8104a70 0x7fb9b819a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:19:52.482 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:51.699+0000 7fb9b659c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb9b8104a70 0x7fb9b819a650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:35058/0 (socket says 172.21.15.142:35058) 2024-09-06T14:19:52.482 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:51.699+0000 7fb9b659c640 1 -- 172.21.15.142:0/484698567 learned_addr learned my addr 172.21.15.142:0/484698567 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:19:52.482 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:51.699+0000 7fb9b659c640 1 -- 172.21.15.142:0/484698567 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9b819abb0 con 0x7fb9b8104a70 2024-09-06T14:19:52.482 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:51.699+0000 7fb9b659c640 1 --2- 172.21.15.142:0/484698567 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb9b8104a70 0x7fb9b819a650 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fb9a8000c00 tx=0x7fb9a8030b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:19:52.482 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:51.699+0000 7fb9af7fe640 1 -- 172.21.15.142:0/484698567 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb9a8037040 con 0x7fb9b8104a70 2024-09-06T14:19:52.482 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:51.699+0000 7fb9bcf95640 1 -- 172.21.15.142:0/484698567 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9b819ae40 con 0x7fb9b8104a70 2024-09-06T14:19:52.482 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:51.699+0000 7fb9af7fe640 1 -- 172.21.15.142:0/484698567 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7fb9a8040070 con 0x7fb9b8104a70 2024-09-06T14:19:52.482 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:51.699+0000 7fb9af7fe640 1 -- 172.21.15.142:0/484698567 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb9a803a390 con 0x7fb9b8104a70 2024-09-06T14:19:52.482 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:51.699+0000 7fb9bcf95640 1 -- 172.21.15.142:0/484698567 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9b819b800 con 0x7fb9b8104a70 2024-09-06T14:19:52.482 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:51.700+0000 7fb9af7fe640 1 -- 172.21.15.142:0/484698567 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 8) ==== 50075+0+0 (secure 0 0 0) 0x7fb9a803b430 con 0x7fb9b8104a70 2024-09-06T14:19:52.483 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:51.700+0000 7fb9bcf95640 1 -- 172.21.15.142:0/484698567 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9b8104e50 con 0x7fb9b8104a70 2024-09-06T14:19:52.483 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:51.700+0000 7fb9af7fe640 1 --2- 172.21.15.142:0/484698567 >> v2:172.21.15.142:6800/3892080990 conn(0x7fb9a403da00 0x7fb9a403fec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:19:52.483 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:51.700+0000 7fb9af7fe640 1 -- 172.21.15.142:0/484698567 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 854+0+0 (secure 0 0 0) 0x7fb9a8071ba0 con 0x7fb9b8104a70 2024-09-06T14:19:52.483 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:51.700+0000 7fb9affff640 1 --2- 172.21.15.142:0/484698567 >> v2:172.21.15.142:6800/3892080990 conn(0x7fb9a403da00 0x7fb9a403fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:19:52.483 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:51.701+0000 7fb9affff640 1 --2- 172.21.15.142:0/484698567 >> v2:172.21.15.142:6800/3892080990 conn(0x7fb9a403da00 0x7fb9a403fec0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fb9a00098c0 tx=0x7fb9a0006db0 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:19:52.483 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:51.705+0000 7fb9af7fe640 1 -- 172.21.15.142:0/484698567 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7fb9a8043360 con 0x7fb9b8104a70 2024-09-06T14:19:52.483 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:51.860+0000 7fb9bcf95640 1 -- 172.21.15.142:0/484698567 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7fb9b819c070 con 0x7fb9b8104a70 2024-09-06T14:19:52.483 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:51.865+0000 7fb9af7fe640 1 -- 172.21.15.142:0/484698567 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7fb9a802e200 con 0x7fb9b8104a70 2024-09-06T14:19:52.483 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:51.867+0000 7fb9bcf95640 1 -- 172.21.15.142:0/484698567 >> v2:172.21.15.142:6800/3892080990 conn(0x7fb9a403da00 msgr2=0x7fb9a403fec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:19:52.483 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:51.867+0000 7fb9bcf95640 1 --2- 172.21.15.142:0/484698567 >> v2:172.21.15.142:6800/3892080990 conn(0x7fb9a403da00 0x7fb9a403fec0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fb9a00098c0 tx=0x7fb9a0006db0 comp rx=0 tx=0).stop 2024-09-06T14:19:52.483 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:51.867+0000 7fb9bcf95640 1 -- 172.21.15.142:0/484698567 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb9b8104a70 msgr2=0x7fb9b819a650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:19:52.483 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:51.867+0000 7fb9bcf95640 1 --2- 172.21.15.142:0/484698567 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb9b8104a70 0x7fb9b819a650 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fb9a8000c00 tx=0x7fb9a8030b70 comp rx=0 tx=0).stop 2024-09-06T14:19:52.483 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:51.868+0000 7fb9bcf95640 1 -- 172.21.15.142:0/484698567 shutdown_connections 2024-09-06T14:19:52.483 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:51.868+0000 7fb9bcf95640 1 --2- 172.21.15.142:0/484698567 >> v2:172.21.15.142:6800/3892080990 conn(0x7fb9a403da00 0x7fb9a403fec0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:19:52.484 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:51.868+0000 7fb9bcf95640 1 --2- 172.21.15.142:0/484698567 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb9b8104a70 0x7fb9b819a650 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:19:52.484 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:51.868+0000 7fb9bcf95640 1 -- 172.21.15.142:0/484698567 >> 172.21.15.142:0/484698567 conn(0x7fb9b8100700 msgr2=0x7fb9b8101220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:19:52.484 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:51.868+0000 7fb9bcf95640 1 -- 172.21.15.142:0/484698567 shutdown_connections 2024-09-06T14:19:52.484 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:51.868+0000 7fb9bcf95640 1 -- 172.21.15.142:0/484698567 wait complete. 2024-09-06T14:19:52.484 INFO:teuthology.orchestra.run.smithi142.stdout:Enabling the dashboard module... 2024-09-06T14:19:53.107 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:52 smithi142 ceph-mon[29958]: from='mgr.14116 172.21.15.142:0/3534094648' entity='mgr.y' 2024-09-06T14:19:53.108 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:52 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/484698567' entity='client.admin' 2024-09-06T14:19:54.098 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:53 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/1003244567' entity='client.admin' cmd={"prefix": "mgr module enable", "module": "dashboard"} : dispatch 2024-09-06T14:19:54.098 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:53 smithi142 ceph-mon[29958]: from='mgr.14116 172.21.15.142:0/3534094648' entity='mgr.y' 2024-09-06T14:19:54.099 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:19:53 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: ignoring --setuser ceph since I am not root 2024-09-06T14:19:54.099 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:19:53 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: ignoring --setgroup ceph since I am not root 2024-09-06T14:19:54.498 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:52.893+0000 7f45aecf8640 1 Processor -- start 2024-09-06T14:19:54.498 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:52.893+0000 7f45aecf8640 1 -- start start 2024-09-06T14:19:54.498 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:52.893+0000 7f45aecf8640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f45a8106710 0x7f45a8106b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:19:54.498 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:52.893+0000 7f45aecf8640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f45a8107100 con 0x7f45a8106710 2024-09-06T14:19:54.498 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:52.893+0000 7f45aca94640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f45a8106710 0x7f45a8106b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:19:54.498 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:52.893+0000 7f45aca94640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f45a8106710 0x7f45a8106b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:35064/0 (socket says 172.21.15.142:35064) 2024-09-06T14:19:54.499 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:52.893+0000 7f45aca94640 1 -- 172.21.15.142:0/3319249880 learned_addr learned my addr 172.21.15.142:0/3319249880 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:19:54.499 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:52.894+0000 7f45aca94640 1 -- 172.21.15.142:0/3319249880 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45a8107280 con 0x7f45a8106710 2024-09-06T14:19:54.499 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:52.894+0000 7f45aca94640 1 --2- 172.21.15.142:0/3319249880 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f45a8106710 0x7f45a8106b10 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f459c009e80 tx=0x7f459c02a7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be47545b80cc0551 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:19:54.499 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:52.894+0000 7f45a77fe640 1 -- 172.21.15.142:0/3319249880 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f459c036070 con 0x7f45a8106710 2024-09-06T14:19:54.499 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:52.895+0000 7f45a77fe640 1 -- 172.21.15.142:0/3319249880 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7f459c031880 con 0x7f45a8106710 2024-09-06T14:19:54.499 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:52.895+0000 7f45a77fe640 1 -- 172.21.15.142:0/3319249880 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f459c007880 con 0x7f45a8106710 2024-09-06T14:19:54.499 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:52.895+0000 7f45aecf8640 1 -- 172.21.15.142:0/3319249880 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f45a8106710 msgr2=0x7f45a8106b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:19:54.499 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:52.895+0000 7f45aecf8640 1 --2- 172.21.15.142:0/3319249880 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f45a8106710 0x7f45a8106b10 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f459c009e80 tx=0x7f459c02a7a0 comp rx=0 tx=0).stop 2024-09-06T14:19:54.499 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:52.895+0000 7f45aecf8640 1 -- 172.21.15.142:0/3319249880 shutdown_connections 2024-09-06T14:19:54.499 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:52.895+0000 7f45aecf8640 1 --2- 172.21.15.142:0/3319249880 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f45a8106710 0x7f45a8106b10 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:19:54.499 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:52.895+0000 7f45aecf8640 1 -- 172.21.15.142:0/3319249880 >> 172.21.15.142:0/3319249880 conn(0x7f45a8102020 msgr2=0x7f45a8104440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:19:54.499 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:52.895+0000 7f45aecf8640 1 -- 172.21.15.142:0/3319249880 shutdown_connections 2024-09-06T14:19:54.499 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:52.895+0000 7f45aecf8640 1 -- 172.21.15.142:0/3319249880 wait complete. 2024-09-06T14:19:54.500 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:52.896+0000 7f45aecf8640 1 Processor -- start 2024-09-06T14:19:54.500 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:52.896+0000 7f45aecf8640 1 -- start start 2024-09-06T14:19:54.500 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:52.896+0000 7f45aecf8640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f45a8106710 0x7f45a807c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:19:54.500 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:52.896+0000 7f45aecf8640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f45a810ab40 con 0x7f45a8106710 2024-09-06T14:19:54.500 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:52.896+0000 7f45aca94640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f45a8106710 0x7f45a807c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:19:54.500 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:52.896+0000 7f45aca94640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f45a8106710 0x7f45a807c200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:35066/0 (socket says 172.21.15.142:35066) 2024-09-06T14:19:54.500 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:52.896+0000 7f45aca94640 1 -- 172.21.15.142:0/1003244567 learned_addr learned my addr 172.21.15.142:0/1003244567 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:19:54.500 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:52.896+0000 7f45aca94640 1 -- 172.21.15.142:0/1003244567 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45a807c760 con 0x7f45a8106710 2024-09-06T14:19:54.500 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:52.896+0000 7f45aca94640 1 --2- 172.21.15.142:0/1003244567 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f45a8106710 0x7f45a807c200 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f459c007db0 tx=0x7f459c031b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:19:54.500 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:52.896+0000 7f45a5ffb640 1 -- 172.21.15.142:0/1003244567 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f459c041020 con 0x7f45a8106710 2024-09-06T14:19:54.501 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:52.896+0000 7f45aecf8640 1 -- 172.21.15.142:0/1003244567 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45a807ab50 con 0x7f45a8106710 2024-09-06T14:19:54.501 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:52.896+0000 7f45a5ffb640 1 -- 172.21.15.142:0/1003244567 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7f459c031530 con 0x7f45a8106710 2024-09-06T14:19:54.501 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:52.896+0000 7f45a5ffb640 1 -- 172.21.15.142:0/1003244567 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f459c036040 con 0x7f45a8106710 2024-09-06T14:19:54.501 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:52.897+0000 7f45aecf8640 1 -- 172.21.15.142:0/1003244567 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45a807b130 con 0x7f45a8106710 2024-09-06T14:19:54.501 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:52.897+0000 7f45a5ffb640 1 -- 172.21.15.142:0/1003244567 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 8) ==== 50075+0+0 (secure 0 0 0) 0x7f459c044020 con 0x7f45a8106710 2024-09-06T14:19:54.501 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:52.898+0000 7f45aecf8640 1 -- 172.21.15.142:0/1003244567 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45a8106b90 con 0x7f45a8106710 2024-09-06T14:19:54.501 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:52.898+0000 7f45a5ffb640 1 --2- 172.21.15.142:0/1003244567 >> v2:172.21.15.142:6800/3892080990 conn(0x7f459803da00 0x7f459803fec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:19:54.501 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:52.898+0000 7f45a5ffb640 1 -- 172.21.15.142:0/1003244567 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 854+0+0 (secure 0 0 0) 0x7f459c072f10 con 0x7f45a8106710 2024-09-06T14:19:54.501 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:52.898+0000 7f45a7fff640 1 --2- 172.21.15.142:0/1003244567 >> v2:172.21.15.142:6800/3892080990 conn(0x7f459803da00 0x7f459803fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:19:54.501 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:52.898+0000 7f45a7fff640 1 --2- 172.21.15.142:0/1003244567 >> v2:172.21.15.142:6800/3892080990 conn(0x7f459803da00 0x7f459803fec0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f45940098c0 tx=0x7f4594006db0 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:19:54.501 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:52.902+0000 7f45a5ffb640 1 -- 172.21.15.142:0/1003244567 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7f459c04b020 con 0x7f45a8106710 2024-09-06T14:19:54.501 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:53.112+0000 7f45aecf8640 1 -- 172.21.15.142:0/1003244567 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7f45a807b600 con 0x7f45a8106710 2024-09-06T14:19:54.501 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:53.857+0000 7f45a5ffb640 1 -- 172.21.15.142:0/1003244567 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7f459c04be00 con 0x7f45a8106710 2024-09-06T14:19:54.501 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:53.859+0000 7f45a5ffb640 1 -- 172.21.15.142:0/1003244567 <== mon.0 v2:172.21.15.142:3300/0 8 ==== mgrmap(e 9) ==== 50088+0+0 (secure 0 0 0) 0x7f459c03c470 con 0x7f45a8106710 2024-09-06T14:19:54.502 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:53.860+0000 7f45aecf8640 1 -- 172.21.15.142:0/1003244567 >> v2:172.21.15.142:6800/3892080990 conn(0x7f459803da00 msgr2=0x7f459803fec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:19:54.502 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:53.860+0000 7f45aecf8640 1 --2- 172.21.15.142:0/1003244567 >> v2:172.21.15.142:6800/3892080990 conn(0x7f459803da00 0x7f459803fec0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f45940098c0 tx=0x7f4594006db0 comp rx=0 tx=0).stop 2024-09-06T14:19:54.502 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:53.860+0000 7f45aecf8640 1 -- 172.21.15.142:0/1003244567 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f45a8106710 msgr2=0x7f45a807c200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:19:54.502 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:53.860+0000 7f45aecf8640 1 --2- 172.21.15.142:0/1003244567 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f45a8106710 0x7f45a807c200 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f459c007db0 tx=0x7f459c031b10 comp rx=0 tx=0).stop 2024-09-06T14:19:54.502 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:53.860+0000 7f45aecf8640 1 -- 172.21.15.142:0/1003244567 shutdown_connections 2024-09-06T14:19:54.502 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:53.860+0000 7f45aecf8640 1 --2- 172.21.15.142:0/1003244567 >> v2:172.21.15.142:6800/3892080990 conn(0x7f459803da00 0x7f459803fec0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:19:54.502 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:53.860+0000 7f45aecf8640 1 --2- 172.21.15.142:0/1003244567 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f45a8106710 0x7f45a807c200 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:19:54.502 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:53.860+0000 7f45aecf8640 1 -- 172.21.15.142:0/1003244567 >> 172.21.15.142:0/1003244567 conn(0x7f45a8102020 msgr2=0x7f45a8102b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:19:54.502 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:53.860+0000 7f45aecf8640 1 -- 172.21.15.142:0/1003244567 shutdown_connections 2024-09-06T14:19:54.502 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:53.860+0000 7f45aecf8640 1 -- 172.21.15.142:0/1003244567 wait complete. 2024-09-06T14:19:55.111 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:54 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/1003244567' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-09-06T14:19:55.111 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:54 smithi142 ceph-mon[29958]: mgrmap e9: y(active, since 17s) 2024-09-06T14:19:55.457 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:19:55 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: 2024-09-06T14:19:55.163+0000 7fadf06b6040 -1 mgr[py] Module not found: 'volumes' 2024-09-06T14:19:55.457 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:19:55 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: 2024-09-06T14:19:55.163+0000 7fadf06b6040 -1 mgr[py] Traceback (most recent call last): 2024-09-06T14:19:55.457 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:19:55 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: File "/usr/share/ceph/mgr/volumes/__init__.py", line 2, in 2024-09-06T14:19:55.457 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:19:55 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: from .module import Module 2024-09-06T14:19:55.457 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:19:55 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: File "/usr/share/ceph/mgr/volumes/module.py", line 9, in 2024-09-06T14:19:55.457 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:19:55 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: from .fs.volume import VolumeClient 2024-09-06T14:19:55.457 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:19:55 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: File "/usr/share/ceph/mgr/volumes/fs/volume.py", line 12, in 2024-09-06T14:19:55.457 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:19:55 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: from ceph.fs.earmarking import CephFSVolumeEarmarking, EarmarkException # type: ignore 2024-09-06T14:19:55.457 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:19:55 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: ModuleNotFoundError: No module named 'ceph.fs' 2024-09-06T14:19:55.457 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:19:55 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: 2024-09-06T14:19:55.458 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:19:55 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: 2024-09-06T14:19:55.164+0000 7fadf06b6040 -1 mgr[py] Class not found in module 'volumes' 2024-09-06T14:19:55.458 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:19:55 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: 2024-09-06T14:19:55.164+0000 7fadf06b6040 -1 mgr[py] Error loading module 'volumes': (2) No such file or directory 2024-09-06T14:19:55.772 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout { 2024-09-06T14:19:55.772 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-09-06T14:19:55.772 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-06T14:19:55.773 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-09-06T14:19:55.773 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-06T14:19:55.773 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout } 2024-09-06T14:19:55.773 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:54.953+0000 7f372c08a640 1 Processor -- start 2024-09-06T14:19:55.773 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:54.953+0000 7f372c08a640 1 -- start start 2024-09-06T14:19:55.773 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:54.953+0000 7f372c08a640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f37241055b0 0x7f37241059b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:19:55.773 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:54.953+0000 7f372c08a640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f3724105fa0 con 0x7f37241055b0 2024-09-06T14:19:55.773 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:54.954+0000 7f3729e26640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f37241055b0 0x7f37241059b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:19:55.773 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:54.954+0000 7f3729e26640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f37241055b0 0x7f37241059b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:35098/0 (socket says 172.21.15.142:35098) 2024-09-06T14:19:55.773 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:54.954+0000 7f3729e26640 1 -- 172.21.15.142:0/968960309 learned_addr learned my addr 172.21.15.142:0/968960309 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:19:55.773 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:54.954+0000 7f3729e26640 1 -- 172.21.15.142:0/968960309 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3724106120 con 0x7f37241055b0 2024-09-06T14:19:55.773 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:54.955+0000 7f3729e26640 1 --2- 172.21.15.142:0/968960309 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f37241055b0 0x7f37241059b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3720009820 tx=0x7f372002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1e279e3afd95981 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:19:55.774 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:54.955+0000 7f3728e24640 1 -- 172.21.15.142:0/968960309 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3720037070 con 0x7f37241055b0 2024-09-06T14:19:55.774 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:54.955+0000 7f3728e24640 1 -- 172.21.15.142:0/968960309 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7f37200328c0 con 0x7f37241055b0 2024-09-06T14:19:55.774 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:54.955+0000 7f3728e24640 1 -- 172.21.15.142:0/968960309 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3720030640 con 0x7f37241055b0 2024-09-06T14:19:55.774 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:54.955+0000 7f372c08a640 1 -- 172.21.15.142:0/968960309 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f37241055b0 msgr2=0x7f37241059b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:19:55.774 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:54.955+0000 7f372c08a640 1 --2- 172.21.15.142:0/968960309 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f37241055b0 0x7f37241059b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3720009820 tx=0x7f372002a270 comp rx=0 tx=0).stop 2024-09-06T14:19:55.774 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:54.956+0000 7f372c08a640 1 -- 172.21.15.142:0/968960309 shutdown_connections 2024-09-06T14:19:55.774 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:54.956+0000 7f372c08a640 1 --2- 172.21.15.142:0/968960309 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f37241055b0 0x7f37241059b0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:19:55.774 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:54.956+0000 7f372c08a640 1 -- 172.21.15.142:0/968960309 >> 172.21.15.142:0/968960309 conn(0x7f3724100f40 msgr2=0x7f3724103360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:19:55.774 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:54.956+0000 7f372c08a640 1 -- 172.21.15.142:0/968960309 shutdown_connections 2024-09-06T14:19:55.774 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:54.956+0000 7f372c08a640 1 -- 172.21.15.142:0/968960309 wait complete. 2024-09-06T14:19:55.774 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:54.956+0000 7f372c08a640 1 Processor -- start 2024-09-06T14:19:55.774 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:54.956+0000 7f372c08a640 1 -- start start 2024-09-06T14:19:55.774 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:54.957+0000 7f372c08a640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f37241055b0 0x7f3724198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:19:55.775 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:54.957+0000 7f372c08a640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f3724078ff0 con 0x7f37241055b0 2024-09-06T14:19:55.775 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:54.957+0000 7f3729e26640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f37241055b0 0x7f3724198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:19:55.775 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:54.957+0000 7f3729e26640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f37241055b0 0x7f3724198490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:35106/0 (socket says 172.21.15.142:35106) 2024-09-06T14:19:55.775 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:54.957+0000 7f3729e26640 1 -- 172.21.15.142:0/3694828588 learned_addr learned my addr 172.21.15.142:0/3694828588 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:19:55.775 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:54.957+0000 7f3729e26640 1 -- 172.21.15.142:0/3694828588 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37241989f0 con 0x7f37241055b0 2024-09-06T14:19:55.775 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:54.957+0000 7f3729e26640 1 --2- 172.21.15.142:0/3694828588 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f37241055b0 0x7f3724198490 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f372002a830 tx=0x7f3720030bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:19:55.775 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:54.957+0000 7f371affd640 1 -- 172.21.15.142:0/3694828588 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3720037040 con 0x7f37241055b0 2024-09-06T14:19:55.775 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:54.957+0000 7f371affd640 1 -- 172.21.15.142:0/3694828588 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7f3720040070 con 0x7f37241055b0 2024-09-06T14:19:55.775 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:54.957+0000 7f371affd640 1 -- 172.21.15.142:0/3694828588 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f372003be40 con 0x7f37241055b0 2024-09-06T14:19:55.775 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:54.957+0000 7f372c08a640 1 -- 172.21.15.142:0/3694828588 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3724198c80 con 0x7f37241055b0 2024-09-06T14:19:55.775 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:54.958+0000 7f372c08a640 1 -- 172.21.15.142:0/3694828588 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3724199640 con 0x7f37241055b0 2024-09-06T14:19:55.775 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:54.959+0000 7f372c08a640 1 -- 172.21.15.142:0/3694828588 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37241059b0 con 0x7f37241055b0 2024-09-06T14:19:55.775 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:54.959+0000 7f371affd640 1 -- 172.21.15.142:0/3694828588 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 9) ==== 50088+0+0 (secure 0 0 0) 0x7f372003b430 con 0x7f37241055b0 2024-09-06T14:19:55.776 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:54.959+0000 7f371affd640 1 --2- 172.21.15.142:0/3694828588 >> v2:172.21.15.142:6800/3892080990 conn(0x7f371003fcf0 0x7f37100421b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:19:55.776 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:54.959+0000 7f371affd640 1 -- 172.21.15.142:0/3694828588 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 854+0+0 (secure 0 0 0) 0x7f3720071b20 con 0x7f37241055b0 2024-09-06T14:19:55.776 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:54.959+0000 7f3729625640 1 -- 172.21.15.142:0/3694828588 >> v2:172.21.15.142:6800/3892080990 conn(0x7f371003fcf0 msgr2=0x7f37100421b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:6800/3892080990 2024-09-06T14:19:55.776 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:54.959+0000 7f3729625640 1 --2- 172.21.15.142:0/3694828588 >> v2:172.21.15.142:6800/3892080990 conn(0x7f371003fcf0 0x7f37100421b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-06T14:19:55.776 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:54.967+0000 7f371affd640 1 -- 172.21.15.142:0/3694828588 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7f372002e200 con 0x7f37241055b0 2024-09-06T14:19:55.776 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:55.161+0000 7f3729625640 1 -- 172.21.15.142:0/3694828588 >> v2:172.21.15.142:6800/3892080990 conn(0x7f371003fcf0 msgr2=0x7f37100421b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:6800/3892080990 2024-09-06T14:19:55.776 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:55.161+0000 7f3729625640 1 --2- 172.21.15.142:0/3694828588 >> v2:172.21.15.142:6800/3892080990 conn(0x7f371003fcf0 0x7f37100421b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-06T14:19:55.776 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:55.174+0000 7f372c08a640 1 -- 172.21.15.142:0/3694828588 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f3724191240 con 0x7f37241055b0 2024-09-06T14:19:55.776 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:55.175+0000 7f371affd640 1 -- 172.21.15.142:0/3694828588 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7f37200328c0 con 0x7f37241055b0 2024-09-06T14:19:55.776 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:55.177+0000 7f372c08a640 1 -- 172.21.15.142:0/3694828588 >> v2:172.21.15.142:6800/3892080990 conn(0x7f371003fcf0 msgr2=0x7f37100421b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-06T14:19:55.776 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:55.177+0000 7f372c08a640 1 --2- 172.21.15.142:0/3694828588 >> v2:172.21.15.142:6800/3892080990 conn(0x7f371003fcf0 0x7f37100421b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:19:55.776 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:55.177+0000 7f372c08a640 1 -- 172.21.15.142:0/3694828588 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f37241055b0 msgr2=0x7f3724198490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:19:55.777 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:55.177+0000 7f372c08a640 1 --2- 172.21.15.142:0/3694828588 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f37241055b0 0x7f3724198490 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f372002a830 tx=0x7f3720030bc0 comp rx=0 tx=0).stop 2024-09-06T14:19:55.777 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:55.177+0000 7f372c08a640 1 -- 172.21.15.142:0/3694828588 shutdown_connections 2024-09-06T14:19:55.777 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:55.177+0000 7f372c08a640 1 --2- 172.21.15.142:0/3694828588 >> v2:172.21.15.142:6800/3892080990 conn(0x7f371003fcf0 0x7f37100421b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:19:55.777 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:55.177+0000 7f372c08a640 1 --2- 172.21.15.142:0/3694828588 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f37241055b0 0x7f3724198490 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:19:55.777 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:55.177+0000 7f372c08a640 1 -- 172.21.15.142:0/3694828588 >> 172.21.15.142:0/3694828588 conn(0x7f3724100f40 msgr2=0x7f372418e920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:19:55.777 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:55.177+0000 7f372c08a640 1 -- 172.21.15.142:0/3694828588 shutdown_connections 2024-09-06T14:19:55.777 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:55.177+0000 7f372c08a640 1 -- 172.21.15.142:0/3694828588 wait complete. 2024-09-06T14:19:55.777 INFO:teuthology.orchestra.run.smithi142.stdout:Waiting for the mgr to restart... 2024-09-06T14:19:55.777 INFO:teuthology.orchestra.run.smithi142.stdout:Waiting for mgr epoch 9... 2024-09-06T14:19:56.177 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:55 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/3694828588' entity='client.admin' cmd={"prefix": "mgr stat"} : dispatch 2024-09-06T14:19:59.207 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:19:58 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: 2024-09-06T14:19:58.840+0000 7fadf06b6040 -1 log_channel(cluster) log [ERR] : Failed to load ceph-mgr modules: volumes 2024-09-06T14:19:59.207 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:58 smithi142 ceph-mon[29958]: Active manager daemon y restarted 2024-09-06T14:19:59.207 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:58 smithi142 ceph-mon[29958]: Activating manager daemon y 2024-09-06T14:19:59.207 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:58 smithi142 ceph-mon[29958]: osdmap e3: 0 total, 0 up, 0 in 2024-09-06T14:19:59.207 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:58 smithi142 ceph-mon[29958]: mgrmap e10: y(active, starting, since 0.00336142s) 2024-09-06T14:19:59.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:58 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-06T14:19:59.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:58 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-06T14:19:59.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:58 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-09-06T14:19:59.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:58 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-09-06T14:19:59.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:58 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-09-06T14:19:59.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:58 smithi142 ceph-mon[29958]: Manager daemon y is now available 2024-09-06T14:20:00.207 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:59 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-06T14:20:00.207 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:59 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-06T14:20:00.207 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:59 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-06T14:20:00.207 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:19:59 smithi142 ceph-mon[29958]: mgrmap e11: y(active, since 1.00553s) 2024-09-06T14:20:00.207 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:20:00 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: 2024-09-06T14:20:00.042+0000 7fad66117640 -1 log_channel(cephadm) log [ERR] : [06/Sep/2024:14:20:00] ENGINE Error in HTTPServer.serve 2024-09-06T14:20:00.207 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:20:00 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: Traceback (most recent call last): 2024-09-06T14:20:00.207 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:20:00 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-06T14:20:00.208 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:20:00 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: self._connections.run(self.expiration_interval) 2024-09-06T14:20:00.208 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:20:00 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-06T14:20:00.208 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:20:00 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: self._run(expiration_interval) 2024-09-06T14:20:00.208 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:20:00 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-06T14:20:00.208 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:20:00 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: new_conn = self._from_server_socket(self.server.socket) 2024-09-06T14:20:00.208 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:20:00 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-06T14:20:00.208 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:20:00 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-06T14:20:00.208 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:20:00 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-06T14:20:00.208 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:20:00 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: s = self.context.wrap_socket( 2024-09-06T14:20:00.208 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:20:00 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-06T14:20:00.208 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:20:00 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: return self.sslsocket_class._create( 2024-09-06T14:20:00.208 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:20:00 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-06T14:20:00.208 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:20:00 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: self.do_handshake() 2024-09-06T14:20:00.208 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:20:00 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-06T14:20:00.209 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:20:00 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: self._sslobj.do_handshake() 2024-09-06T14:20:00.209 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:20:00 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-06T14:20:00.209 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:20:00 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: 2024-09-06T14:20:00.630 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout { 2024-09-06T14:20:00.630 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-09-06T14:20:00.630 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-06T14:20:00.631 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout } 2024-09-06T14:20:00.631 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:56.182+0000 7fb438b49640 1 Processor -- start 2024-09-06T14:20:00.631 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:56.182+0000 7fb438b49640 1 -- start start 2024-09-06T14:20:00.631 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:56.183+0000 7fb438b49640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb434079c00 0x7fb4341092b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:20:00.631 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:56.183+0000 7fb438b49640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fb4341097f0 con 0x7fb434079c00 2024-09-06T14:20:00.631 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:56.183+0000 7fb43259c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb434079c00 0x7fb4341092b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:20:00.631 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:56.183+0000 7fb43259c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb434079c00 0x7fb4341092b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:35120/0 (socket says 172.21.15.142:35120) 2024-09-06T14:20:00.631 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:56.183+0000 7fb43259c640 1 -- 172.21.15.142:0/3659475923 learned_addr learned my addr 172.21.15.142:0/3659475923 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:20:00.631 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:56.183+0000 7fb43259c640 1 -- 172.21.15.142:0/3659475923 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb434109970 con 0x7fb434079c00 2024-09-06T14:20:00.631 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:56.184+0000 7fb43259c640 1 --2- 172.21.15.142:0/3659475923 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb434079c00 0x7fb4341092b0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb424009820 tx=0x7fb42402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3bbe6573b327369 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:20:00.631 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:56.184+0000 7fb43159a640 1 -- 172.21.15.142:0/3659475923 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb424037070 con 0x7fb434079c00 2024-09-06T14:20:00.631 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:56.184+0000 7fb43159a640 1 -- 172.21.15.142:0/3659475923 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7fb4240328c0 con 0x7fb434079c00 2024-09-06T14:20:00.631 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:56.184+0000 7fb43159a640 1 -- 172.21.15.142:0/3659475923 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb424030640 con 0x7fb434079c00 2024-09-06T14:20:00.631 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:56.185+0000 7fb438b49640 1 -- 172.21.15.142:0/3659475923 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb434079c00 msgr2=0x7fb4341092b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:20:00.632 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:56.185+0000 7fb438b49640 1 --2- 172.21.15.142:0/3659475923 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb434079c00 0x7fb4341092b0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb424009820 tx=0x7fb42402a270 comp rx=0 tx=0).stop 2024-09-06T14:20:00.632 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:56.185+0000 7fb438b49640 1 -- 172.21.15.142:0/3659475923 shutdown_connections 2024-09-06T14:20:00.632 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:56.185+0000 7fb438b49640 1 --2- 172.21.15.142:0/3659475923 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb434079c00 0x7fb4341092b0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:20:00.632 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:56.185+0000 7fb438b49640 1 -- 172.21.15.142:0/3659475923 >> 172.21.15.142:0/3659475923 conn(0x7fb434100f40 msgr2=0x7fb434103360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:20:00.632 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:56.185+0000 7fb438b49640 1 -- 172.21.15.142:0/3659475923 shutdown_connections 2024-09-06T14:20:00.632 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:56.185+0000 7fb438b49640 1 -- 172.21.15.142:0/3659475923 wait complete. 2024-09-06T14:20:00.632 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:56.186+0000 7fb438b49640 1 Processor -- start 2024-09-06T14:20:00.632 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:56.186+0000 7fb438b49640 1 -- start start 2024-09-06T14:20:00.632 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:56.186+0000 7fb438b49640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb434079c00 0x7fb43419a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:20:00.632 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:56.186+0000 7fb438b49640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fb43410aa20 con 0x7fb434079c00 2024-09-06T14:20:00.632 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:56.186+0000 7fb43259c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb434079c00 0x7fb43419a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:20:00.632 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:56.186+0000 7fb43259c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb434079c00 0x7fb43419a6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:35134/0 (socket says 172.21.15.142:35134) 2024-09-06T14:20:00.633 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:56.186+0000 7fb43259c640 1 -- 172.21.15.142:0/3428598608 learned_addr learned my addr 172.21.15.142:0/3428598608 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:20:00.633 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:56.186+0000 7fb43259c640 1 -- 172.21.15.142:0/3428598608 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb43419ac20 con 0x7fb434079c00 2024-09-06T14:20:00.633 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:56.186+0000 7fb43259c640 1 --2- 172.21.15.142:0/3428598608 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb434079c00 0x7fb43419a6c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb42402a830 tx=0x7fb424030bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:20:00.633 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:56.187+0000 7fb4237fe640 1 -- 172.21.15.142:0/3428598608 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb424037040 con 0x7fb434079c00 2024-09-06T14:20:00.633 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:56.187+0000 7fb4237fe640 1 -- 172.21.15.142:0/3428598608 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7fb424040070 con 0x7fb434079c00 2024-09-06T14:20:00.633 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:56.187+0000 7fb438b49640 1 -- 172.21.15.142:0/3428598608 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb43419aeb0 con 0x7fb434079c00 2024-09-06T14:20:00.633 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:56.187+0000 7fb4237fe640 1 -- 172.21.15.142:0/3428598608 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb42403be10 con 0x7fb434079c00 2024-09-06T14:20:00.633 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:56.187+0000 7fb438b49640 1 -- 172.21.15.142:0/3428598608 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb43419b870 con 0x7fb434079c00 2024-09-06T14:20:00.633 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:56.188+0000 7fb4237fe640 1 -- 172.21.15.142:0/3428598608 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 9) ==== 50088+0+0 (secure 0 0 0) 0x7fb424045460 con 0x7fb434079c00 2024-09-06T14:20:00.633 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:56.188+0000 7fb4237fe640 1 --2- 172.21.15.142:0/3428598608 >> v2:172.21.15.142:6800/3892080990 conn(0x7fb41c03da50 0x7fb41c03ff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:20:00.633 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:56.188+0000 7fb4237fe640 1 -- 172.21.15.142:0/3428598608 --> v2:172.21.15.142:6800/3892080990 -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fb41c0405e0 con 0x7fb41c03da50 2024-09-06T14:20:00.633 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:56.188+0000 7fb431d9b640 1 -- 172.21.15.142:0/3428598608 >> v2:172.21.15.142:6800/3892080990 conn(0x7fb41c03da50 msgr2=0x7fb41c03ff10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:6800/3892080990 2024-09-06T14:20:00.634 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:56.188+0000 7fb4237fe640 1 -- 172.21.15.142:0/3428598608 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 854+0+0 (secure 0 0 0) 0x7fb424071b40 con 0x7fb434079c00 2024-09-06T14:20:00.634 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:56.188+0000 7fb431d9b640 1 --2- 172.21.15.142:0/3428598608 >> v2:172.21.15.142:6800/3892080990 conn(0x7fb41c03da50 0x7fb41c03ff10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-06T14:20:00.634 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:56.389+0000 7fb431d9b640 1 -- 172.21.15.142:0/3428598608 >> v2:172.21.15.142:6800/3892080990 conn(0x7fb41c03da50 msgr2=0x7fb41c03ff10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:6800/3892080990 2024-09-06T14:20:00.634 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:56.389+0000 7fb431d9b640 1 --2- 172.21.15.142:0/3428598608 >> v2:172.21.15.142:6800/3892080990 conn(0x7fb41c03da50 0x7fb41c03ff10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-06T14:20:00.634 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:56.790+0000 7fb431d9b640 1 -- 172.21.15.142:0/3428598608 >> v2:172.21.15.142:6800/3892080990 conn(0x7fb41c03da50 msgr2=0x7fb41c03ff10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:6800/3892080990 2024-09-06T14:20:00.634 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:56.790+0000 7fb431d9b640 1 --2- 172.21.15.142:0/3428598608 >> v2:172.21.15.142:6800/3892080990 conn(0x7fb41c03da50 0x7fb41c03ff10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-06T14:20:00.634 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:57.591+0000 7fb431d9b640 1 -- 172.21.15.142:0/3428598608 >> v2:172.21.15.142:6800/3892080990 conn(0x7fb41c03da50 msgr2=0x7fb41c03ff10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:6800/3892080990 2024-09-06T14:20:00.634 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:57.591+0000 7fb431d9b640 1 --2- 172.21.15.142:0/3428598608 >> v2:172.21.15.142:6800/3892080990 conn(0x7fb41c03da50 0x7fb41c03ff10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-06T14:20:00.634 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:58.843+0000 7fb4237fe640 1 -- 172.21.15.142:0/3428598608 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mgrmap(e 10) ==== 49941+0+0 (secure 0 0 0) 0x7fb42402ed40 con 0x7fb434079c00 2024-09-06T14:20:00.634 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:58.843+0000 7fb4237fe640 1 -- 172.21.15.142:0/3428598608 >> v2:172.21.15.142:6800/3892080990 conn(0x7fb41c03da50 msgr2=0x7fb41c03ff10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-06T14:20:00.634 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:58.843+0000 7fb4237fe640 1 --2- 172.21.15.142:0/3428598608 >> v2:172.21.15.142:6800/3892080990 conn(0x7fb41c03da50 0x7fb41c03ff10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:20:00.634 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:59.844+0000 7fb4237fe640 1 -- 172.21.15.142:0/3428598608 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mgrmap(e 11) ==== 50033+0+0 (secure 0 0 0) 0x7fb424045750 con 0x7fb434079c00 2024-09-06T14:20:00.634 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:59.845+0000 7fb4237fe640 1 --2- 172.21.15.142:0/3428598608 >> v2:172.21.15.142:6800/853729507 conn(0x7fb41c041460 0x7fb41c043850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:20:00.634 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:59.845+0000 7fb4237fe640 1 -- 172.21.15.142:0/3428598608 --> v2:172.21.15.142:6800/853729507 -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fb424007e50 con 0x7fb41c041460 2024-09-06T14:20:00.635 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:59.845+0000 7fb431d9b640 1 --2- 172.21.15.142:0/3428598608 >> v2:172.21.15.142:6800/853729507 conn(0x7fb41c041460 0x7fb41c043850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:20:00.635 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:59.845+0000 7fb431d9b640 1 --2- 172.21.15.142:0/3428598608 >> v2:172.21.15.142:6800/853729507 conn(0x7fb41c041460 0x7fb41c043850 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fb428003980 tx=0x7fb4280092b0 comp rx=0 tx=0).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:20:00.635 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:59.846+0000 7fb4237fe640 1 -- 172.21.15.142:0/3428598608 <== mgr.14146 v2:172.21.15.142:6800/853729507 1 ==== command_reply(tid 0: 0 ) ==== 8+0+9564 (secure 0 0 0) 0x7fb424007e50 con 0x7fb41c041460 2024-09-06T14:20:00.635 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:59.853+0000 7fb438b49640 1 -- 172.21.15.142:0/3428598608 --> v2:172.21.15.142:6800/853729507 -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7fb4341072a0 con 0x7fb41c041460 2024-09-06T14:20:00.635 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:59.853+0000 7fb4237fe640 1 -- 172.21.15.142:0/3428598608 <== mgr.14146 v2:172.21.15.142:6800/853729507 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7fb4341072a0 con 0x7fb41c041460 2024-09-06T14:20:00.635 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:59.854+0000 7fb438b49640 1 -- 172.21.15.142:0/3428598608 >> v2:172.21.15.142:6800/853729507 conn(0x7fb41c041460 msgr2=0x7fb41c043850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:20:00.635 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:59.854+0000 7fb438b49640 1 --2- 172.21.15.142:0/3428598608 >> v2:172.21.15.142:6800/853729507 conn(0x7fb41c041460 0x7fb41c043850 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fb428003980 tx=0x7fb4280092b0 comp rx=0 tx=0).stop 2024-09-06T14:20:00.635 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:59.854+0000 7fb438b49640 1 -- 172.21.15.142:0/3428598608 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb434079c00 msgr2=0x7fb43419a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:20:00.635 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:59.854+0000 7fb438b49640 1 --2- 172.21.15.142:0/3428598608 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb434079c00 0x7fb43419a6c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb42402a830 tx=0x7fb424030bc0 comp rx=0 tx=0).stop 2024-09-06T14:20:00.635 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:59.854+0000 7fb438b49640 1 -- 172.21.15.142:0/3428598608 shutdown_connections 2024-09-06T14:20:00.635 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:59.854+0000 7fb438b49640 1 --2- 172.21.15.142:0/3428598608 >> v2:172.21.15.142:6800/853729507 conn(0x7fb41c041460 0x7fb41c043850 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:20:00.635 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:59.854+0000 7fb438b49640 1 --2- 172.21.15.142:0/3428598608 >> v2:172.21.15.142:6800/3892080990 conn(0x7fb41c03da50 0x7fb41c03ff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:20:00.635 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:59.854+0000 7fb438b49640 1 --2- 172.21.15.142:0/3428598608 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb434079c00 0x7fb43419a6c0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:20:00.636 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:59.854+0000 7fb438b49640 1 -- 172.21.15.142:0/3428598608 >> 172.21.15.142:0/3428598608 conn(0x7fb434100f40 msgr2=0x7fb434101ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:20:00.636 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:59.854+0000 7fb438b49640 1 -- 172.21.15.142:0/3428598608 shutdown_connections 2024-09-06T14:20:00.636 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:19:59.854+0000 7fb438b49640 1 -- 172.21.15.142:0/3428598608 wait complete. 2024-09-06T14:20:00.636 INFO:teuthology.orchestra.run.smithi142.stdout:mgr epoch 9 is available 2024-09-06T14:20:00.636 INFO:teuthology.orchestra.run.smithi142.stdout:Using certmgr to generate dashboard self-signed certificate... 2024-09-06T14:20:01.175 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:00 smithi142 ceph-mon[29958]: [06/Sep/2024:14:19:59] ENGINE Bus STARTING 2024-09-06T14:20:01.957 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:01 smithi142 ceph-mon[29958]: [06/Sep/2024:14:20:00] ENGINE Serving on https://172.21.15.142:7150 2024-09-06T14:20:01.957 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:01 smithi142 ceph-mon[29958]: [06/Sep/2024:14:20:00] ENGINE Error in HTTPServer.serve 2024-09-06T14:20:01.957 INFO:journalctl@ceph.mon.a.smithi142.stdout: Traceback (most recent call last): 2024-09-06T14:20:01.957 INFO:journalctl@ceph.mon.a.smithi142.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-06T14:20:01.957 INFO:journalctl@ceph.mon.a.smithi142.stdout: self._connections.run(self.expiration_interval) 2024-09-06T14:20:01.957 INFO:journalctl@ceph.mon.a.smithi142.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-06T14:20:01.957 INFO:journalctl@ceph.mon.a.smithi142.stdout: self._run(expiration_interval) 2024-09-06T14:20:01.957 INFO:journalctl@ceph.mon.a.smithi142.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-06T14:20:01.957 INFO:journalctl@ceph.mon.a.smithi142.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-06T14:20:01.957 INFO:journalctl@ceph.mon.a.smithi142.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-06T14:20:01.957 INFO:journalctl@ceph.mon.a.smithi142.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-06T14:20:01.958 INFO:journalctl@ceph.mon.a.smithi142.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-06T14:20:01.958 INFO:journalctl@ceph.mon.a.smithi142.stdout: s = self.context.wrap_socket( 2024-09-06T14:20:01.958 INFO:journalctl@ceph.mon.a.smithi142.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-06T14:20:01.958 INFO:journalctl@ceph.mon.a.smithi142.stdout: return self.sslsocket_class._create( 2024-09-06T14:20:01.958 INFO:journalctl@ceph.mon.a.smithi142.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-06T14:20:01.958 INFO:journalctl@ceph.mon.a.smithi142.stdout: self.do_handshake() 2024-09-06T14:20:01.958 INFO:journalctl@ceph.mon.a.smithi142.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-06T14:20:01.958 INFO:journalctl@ceph.mon.a.smithi142.stdout: self._sslobj.do_handshake() 2024-09-06T14:20:01.958 INFO:journalctl@ceph.mon.a.smithi142.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-06T14:20:01.958 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:01 smithi142 ceph-mon[29958]: [06/Sep/2024:14:20:00] ENGINE Serving on http://172.21.15.142:8765 2024-09-06T14:20:01.958 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:01 smithi142 ceph-mon[29958]: [06/Sep/2024:14:20:00] ENGINE Bus STARTED 2024-09-06T14:20:01.958 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:01 smithi142 ceph-mon[29958]: mgrmap e12: y(active, since 2s) 2024-09-06T14:20:01.958 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:01 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:02.769 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout {"cert": "-----BEGIN CERTIFICATE-----\nMIIE/DCCAuSgAwIBAgIUb9R/oQo1TwhRhSqcw3Or4Cj9wQswDQYJKoZIhvcNAQEL\nBQAwFzEVMBMGA1UEAwwMY2VwaGFkbS1yb290MB4XDTI0MDkwNjE0MjAwMloXDTM0\nMDkwNzE0MjAwMlowGDEWMBQGA1UEAwwNMTcyLjIxLjE1LjE0MjCCAiIwDQYJKoZI\nhvcNAQEBBQADggIPADCCAgoCggIBAOj1nECGL/my9HsXkAsFL7/P+Q6WGU8gS74c\nvCRUvXV1UKbVSk506Vy2ufxSlQPxGVYzB0Wwt9RuoazM745sCp+BZLKO1iqbGGAB\n975SO8iG9WSM7re4dDontHCaLVnXsH0zsbni8ZpbJ8GlMwUxTL7fHkCJgHtxpua1\nwXIX7ErqnoQXMMrQqIsYgDCzZ7hn6Q6WpMBVU2Dn8vkpOhPW/2aqRYKiBnK7RFEV\nIM749CKSQEtoFlsq0YY/HrXe+oXxN/1++ykMzKfKwxxnDo/xUYCCSmBhCXrhtb2M\nlLwM2NQTgf1Px59iwKkCGtwM9xAUlYQ4Y/uDXyrav+ufXszKTXEdkdABbGsznO9n\nWBICVcCxlSb7MNBsWpXXBg1cB1igwvAa6gRC6W2+SC6KChBQueef2VLCxi73xp56\n+JgjrXcJwHbkvwNqpKVquq4g0WBXWVNSZnqG6OYS9AWwiyoqdET2LSQajxWAOuh8\nDNR4kg9MPHBGhMoYYNNztR4AkPFsh3/mi+3+7r2FzAIIw9447814AtHpndvmNudv\nCphLW6zRGvglY6NLUBhhROKQamxhPslJdA1D1+QXJnL00oA7qw4CUUV2TOdrEDQL\npf7BzJosDTP/aSkRW4OBJWMZTVM4TjrUj4CBZkb5AJPAqmHrebXIeCeJgC+gizce\nY9Yx/RcJAgMBAAGjPzA9MC0GA1UdEQQmMCSCCXNtaXRoaTE0MoIRZGFzaGJvYXJk\nX3NlcnZlcnOHBKwVD44wDAYDVR0TAQH/BAIwADANBgkqhkiG9w0BAQsFAAOCAgEA\nLbnKGM4D1CqU4fp56IZ7JI6M2obvJhDK1CJBWjjehQy+s+d/LA3VLd1zYzZ3ugrv\nDVJJT+kgxcvm4OLdkus3Ss43bwDmJp8JJVBuhXj2U8HjYViUdbJQHzEtu4//krkU\n2fVP5OX3SISXg6jYJdlYdn+IfFKeV3OObGoZIWegcbv0f1En0SMnSOomMiUUYhuF\nAI3nmiOYtLC3ukr+hmDUT35KvcYF9C8UOhWZ+Vcf9MBmdPTo2ReAavFAtK2LGG5T\nKfuiiXfkPyZmwAXTE2MJ9CDajT6RKWhDjvPGGV+PpDi6+5u9w2Kz7AR0qbe0cpLR\nEnBbv9fiwnK3R3PQBO0GXRb24txC1+SOQceUddJYIEEd9alxjcPE18xR22NUvamC\nABbv20u2QNfHHRqu3qI+jIPmDRDPUU1C0uyBHi6NBDmKCWtvy9cgbxNgrgb+0UNg\nFqhNi+aad2hzNxCMP/8nwK+IwHgzKhL5dgRKcwXC7W9a+n4iYEn+FdXMBII68EE0\nqiRIKZAgb7dqjcSMWCvLLIWc3eEJyXiF4hQNyRdRpo1HEAEgB5aOX4feqpQAI4f/\n8WzhfeQhxfEf/yHJ3ahcmmo1asBl771Tn0bor14ANlFG1+oNwwz9vpYfFhTY10r9\nhL/qDOg2k4wj1YcAuMKFnsXh6clPdWeaFQXrsIOYERQ=\n-----END CERTIFICATE-----\n", "key": "-----BEGIN RSA PRIVATE KEY-----\nMIIJKAIBAAKCAgEA6PWcQIYv+bL0exeQCwUvv8/5DpYZTyBLvhy8JFS9dXVQptVK\nTnTpXLa5/FKVA/EZVjMHRbC31G6hrMzvjmwKn4Fkso7WKpsYYAH3vlI7yIb1ZIzu\nt7h0Oie0cJotWdewfTOxueLxmlsnwaUzBTFMvt8eQImAe3Gm5rXBchfsSuqehBcw\nytCoixiAMLNnuGfpDpakwFVTYOfy+Sk6E9b/ZqpFgqIGcrtEURUgzvj0IpJAS2gW\nWyrRhj8etd76hfE3/X77KQzMp8rDHGcOj/FRgIJKYGEJeuG1vYyUvAzY1BOB/U/H\nn2LAqQIa3Az3EBSVhDhj+4NfKtq/659ezMpNcR2R0AFsazOc72dYEgJVwLGVJvsw\n0GxaldcGDVwHWKDC8BrqBELpbb5ILooKEFC555/ZUsLGLvfGnnr4mCOtdwnAduS/\nA2qkpWq6riDRYFdZU1Jmeobo5hL0BbCLKip0RPYtJBqPFYA66HwM1HiSD0w8cEaE\nyhhg03O1HgCQ8WyHf+aL7f7uvYXMAgjD3jjvzXgC0emd2+Y2528KmEtbrNEa+CVj\no0tQGGFE4pBqbGE+yUl0DUPX5BcmcvTSgDurDgJRRXZM52sQNAul/sHMmiwNM/9p\nKRFbg4ElYxlNUzhOOtSPgIFmRvkAk8CqYet5tch4J4mAL6CLNx5j1jH9FwkCAwEA\nAQKCAgAIaijC9uXgO9vTX2pDT6RmJg29vmO7JTQ/MXwFRu9SARMQ96bXtybfw7Tr\nnZMG4axT3gN++69kM7vQyVVE3He0w8B+p6SqRNBoVWHcxM0mhONzxphv1/CQ0jLV\nF2iYdG9KUnXyvDUUaYMB3mT9pMZ8JYSK+byDmPsZncqXQ7B0E033APxw8wEHhzLI\nL47JTS2QqpXYYayhlTdEUz4nDc7LZ06b9ZzMg6Gn2R7Ggx0Ua5p5btDR6bXIRtrl\n4GBFus0WMcJCPRVeYTdTfRB8WA/k16/2k8K7JtCBAIU11rP1UyAQpZNVGUTSxgHw\n3WIGOWweKshHwzrKijzA9tB+jrcl03HEFrNU7XSJvPQ0rf4+Onnc4z/ap90FLBh3\nfbFSec82y5fUCSBA+pDyXnpIAgXHdiKDUHQ0697cn6lsgn/I9cu2chpo31l/Qd+X\nDym3Mb3ml/l878fMMA7BYbse+k7TZsTzEvBiuW6ODQD/HMbJaczX24khy8FrKzdo\nhI8ngACAOSBl70uFzqpcKGnEenFiuJKM3ap4pY0/AWlb7b2BRj8BlWRS7xYiLl6G\nG+fcEMy/kX2lGlJE7OYLwY31gUTpILimJZ2QgtgsQGA94MQGajK8UIUtVAPkVMDW\nRjEoyTEMSRDBxCIaV1UwSmA4E8rPCZBHI64cNbG1HAvHQy4V5QKCAQEA+HPrAix+\nOuZX91wV6TCqlTT91JLw3apBZbUGUxSxgizylvCkjiHPnkDhAnMXx7rIPx7HP8fD\nV0ShV9o7IPDZizOf7C+Efdc15EWkpSGdz4afRPx/PcmGSLRgUidS2A93XyLxaGD6\nrT1bsfP5cf2ehH+KVz8hfJTxn1iEdy99w4UlKdFOkVILT28pi1S13diPcAGVwSp5\nRUs7Z6cGxROi9yw45kg3+ztLFskwQhWwJ8g3hAhsi/OBCmiYf6qLehlhzXwaoa/J\nZo3s94wz/Dlqae42YfQFjrTkp18JaFyhACkvbvZvLqWfOZS/GsJQa/Dp/nnRt73G\nod/i85r9M5rEFQKCAQEA8Ak1bnuUUV3d64sD/4YDFMTNTH6TiWzDmbg9cLsMpqKv\nieiZDbj651fLeDeeLScVOMXZ8g3DWV+gkJSgHxCzRrrh2/ILDKodmCerIOznHUP5\nJE3mtaUmU/lcKGZOST/8KXsPwEUfFqhIFe1C+Z0ZM9o95P/szfOGRb2a60IHsK/s\n9X2YyJdpw39VzCSUBA0IPCd4UNwIhDhrelT1oGerJUAdla9gdEZNW7WwsQOomz51\nMhuC3+ssVq/i4cMGmhsv1BHEsEFKa36eEuKH6XYvVB+/DdhuBtTChZ8zKY2c9VuT\n6hFXFkLVm2gWHw1NKh09aAlYA4gv+IqHX0XTrBPAJQKCAQAXtI+utGoS2pl4C3w3\ntKIB/H1fg+mhbLYyqxQAZ1AMeJdo2WsJ+KXjlpMhdbPGJccnZptiNYAo2m5+gF71\n5KUNCA7xLRN6IkeNESwFaXCyNUkK/wBw98JtO3CZRhE8djVQiXf5Go+wadNO5lVd\nj/JI+BZN/sMGLPvk1UBGUuyv9As4szTI3sgfex6+oYV8tkXxTOVp+dB6URRI7oHR\n2e7JoJmJN0jIGr8uXGHTXFhjY1IQsfERZVlhkdtDCxWlv7g6L+YmmSwhMddzQxhO\nr3e8xbLAo9TH6fCLG6E9KOFfr45MCw1OqRTJHK0jM3kS43igBCWDX4MNPNxdm+Me\nwlyFAoIBAQCKFmIcqw9KN9vhE1o7d5byLb2FUhQCLQXhuvHtlk3S1yhxB9R05oPA\n7urov+ZJR4CX5opZC73j/nVad4YeGPdpqz0f0QC2Sm84ByoViPP0DBSZ5A7jVJzW\nFlAEFcbXsO2Z12faTmlyGWFO6+1puAy1ncmZoflTCdOm70reKGBQLwT8vUJVs3Vu\nuCRdinV0VyLzqCk5KzvjBKof+BxOKRV2GopqgDfW7FK4fa+mRdqLZMT7DISf2KTr\nQcgd5Jo5sYY0U/x5ATaU3ISHjLZvgkP7+z9n8ahqWZ+GS22dbBBtvj87mrok3lIQ\nUzLWrkpD/gpDfYyai5A9RURXdM4u4jXZAoIBABjK/Qc9tjok5nUbpYKkXKLUqcco\n0LqiKtO4JpAESmSh72aq+5agDjiF9xotH4Oe0GfLLUGBjzkZxYPG2ulOdaj06cBf\nhpitrVmlZUj2CuKZCG0bFwgTCHua0bnlPkIQlF2A30Atx+akzUGMpXZ3igZaLuoX\n9IUmwe5N6Oqp4jNzFC01KQCoM1Re92jBEDT6prtxeXSTkETa4BCULQ+rxKB/mjn8\n2sQXZPYBV3RSnTdMvCRxCxj0+ibDjt4lxXhylIduBc6tkry+DFnAa9QFJDzJi7oM\neXwnh/hXeKr5xFuRNeW1rzP9WNR57q0Y3NiroA+6Xc8YcF0uEjGf9+SKKGQ=\n-----END RSA PRIVATE KEY-----\n"} 2024-09-06T14:20:02.770 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:01.058+0000 7f73a2405640 1 Processor -- start 2024-09-06T14:20:02.770 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:01.059+0000 7f73a2405640 1 -- start start 2024-09-06T14:20:02.770 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:01.059+0000 7f73a2405640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f739c105b60 0x7f739c105f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:20:02.770 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:01.059+0000 7f73a2405640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f739c1064a0 con 0x7f739c105b60 2024-09-06T14:20:02.770 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:01.059+0000 7f739bfff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f739c105b60 0x7f739c105f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:20:02.770 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:01.059+0000 7f739bfff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f739c105b60 0x7f739c105f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:33520/0 (socket says 172.21.15.142:33520) 2024-09-06T14:20:02.770 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:01.059+0000 7f739bfff640 1 -- 172.21.15.142:0/3955028233 learned_addr learned my addr 172.21.15.142:0/3955028233 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:20:02.770 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:01.059+0000 7f739bfff640 1 -- 172.21.15.142:0/3955028233 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f739c106620 con 0x7f739c105b60 2024-09-06T14:20:02.771 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:01.060+0000 7f739bfff640 1 --2- 172.21.15.142:0/3955028233 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f739c105b60 0x7f739c105f60 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f738c009820 tx=0x7f738c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ca38f0fb97e52b8 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:20:02.771 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:01.060+0000 7f739affd640 1 -- 172.21.15.142:0/3955028233 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f738c037070 con 0x7f739c105b60 2024-09-06T14:20:02.771 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:01.060+0000 7f739affd640 1 -- 172.21.15.142:0/3955028233 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7f738c0328c0 con 0x7f739c105b60 2024-09-06T14:20:02.771 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:01.060+0000 7f739affd640 1 -- 172.21.15.142:0/3955028233 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f738c030640 con 0x7f739c105b60 2024-09-06T14:20:02.771 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:01.061+0000 7f73a2405640 1 -- 172.21.15.142:0/3955028233 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f739c105b60 msgr2=0x7f739c105f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:20:02.771 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:01.061+0000 7f73a2405640 1 --2- 172.21.15.142:0/3955028233 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f739c105b60 0x7f739c105f60 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f738c009820 tx=0x7f738c02a270 comp rx=0 tx=0).stop 2024-09-06T14:20:02.771 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:01.061+0000 7f73a2405640 1 -- 172.21.15.142:0/3955028233 shutdown_connections 2024-09-06T14:20:02.771 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:01.061+0000 7f73a2405640 1 --2- 172.21.15.142:0/3955028233 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f739c105b60 0x7f739c105f60 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:20:02.771 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:01.061+0000 7f73a2405640 1 -- 172.21.15.142:0/3955028233 >> 172.21.15.142:0/3955028233 conn(0x7f739c100700 msgr2=0x7f739c102b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:20:02.771 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:01.061+0000 7f73a2405640 1 -- 172.21.15.142:0/3955028233 shutdown_connections 2024-09-06T14:20:02.771 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:01.061+0000 7f73a2405640 1 -- 172.21.15.142:0/3955028233 wait complete. 2024-09-06T14:20:02.771 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:01.061+0000 7f73a2405640 1 Processor -- start 2024-09-06T14:20:02.772 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:01.061+0000 7f73a2405640 1 -- start start 2024-09-06T14:20:02.772 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:01.062+0000 7f73a2405640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f739c105b60 0x7f739c190540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:20:02.772 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:01.062+0000 7f73a2405640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f739c107720 con 0x7f739c105b60 2024-09-06T14:20:02.772 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:01.062+0000 7f739bfff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f739c105b60 0x7f739c190540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:20:02.772 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:01.062+0000 7f739bfff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f739c105b60 0x7f739c190540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:33534/0 (socket says 172.21.15.142:33534) 2024-09-06T14:20:02.773 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:01.062+0000 7f739bfff640 1 -- 172.21.15.142:0/2189578587 learned_addr learned my addr 172.21.15.142:0/2189578587 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:20:02.773 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:01.062+0000 7f739bfff640 1 -- 172.21.15.142:0/2189578587 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f739c190aa0 con 0x7f739c105b60 2024-09-06T14:20:02.773 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:01.062+0000 7f739bfff640 1 --2- 172.21.15.142:0/2189578587 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f739c105b60 0x7f739c190540 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f738c006f10 tx=0x7f738c032740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:20:02.773 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:01.062+0000 7f73997fa640 1 -- 172.21.15.142:0/2189578587 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f738c037040 con 0x7f739c105b60 2024-09-06T14:20:02.773 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:01.062+0000 7f73997fa640 1 -- 172.21.15.142:0/2189578587 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7f738c040070 con 0x7f739c105b60 2024-09-06T14:20:02.773 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:01.062+0000 7f73a2405640 1 -- 172.21.15.142:0/2189578587 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f739c18ee90 con 0x7f739c105b60 2024-09-06T14:20:02.774 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:01.062+0000 7f73997fa640 1 -- 172.21.15.142:0/2189578587 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f738c03bde0 con 0x7f739c105b60 2024-09-06T14:20:02.774 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:01.062+0000 7f73a2405640 1 -- 172.21.15.142:0/2189578587 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f739c18f410 con 0x7f739c105b60 2024-09-06T14:20:02.774 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:01.063+0000 7f73997fa640 1 -- 172.21.15.142:0/2189578587 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 12) ==== 50088+0+0 (secure 0 0 0) 0x7f738c045460 con 0x7f739c105b60 2024-09-06T14:20:02.774 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:01.063+0000 7f73a2405640 1 -- 172.21.15.142:0/2189578587 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f739c105f60 con 0x7f739c105b60 2024-09-06T14:20:02.774 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:01.063+0000 7f73997fa640 1 --2- 172.21.15.142:0/2189578587 >> v2:172.21.15.142:6800/853729507 conn(0x7f738803da50 0x7f738803ff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:20:02.774 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:01.064+0000 7f73997fa640 1 -- 172.21.15.142:0/2189578587 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 983+0+0 (secure 0 0 0) 0x7f738c071e20 con 0x7f739c105b60 2024-09-06T14:20:02.774 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:01.064+0000 7f739b7fe640 1 --2- 172.21.15.142:0/2189578587 >> v2:172.21.15.142:6800/853729507 conn(0x7f738803da50 0x7f738803ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:20:02.774 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:01.064+0000 7f739b7fe640 1 --2- 172.21.15.142:0/2189578587 >> v2:172.21.15.142:6800/853729507 conn(0x7f738803da50 0x7f738803ff10 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f73840098c0 tx=0x7f7384006db0 comp rx=0 tx=0).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:20:02.774 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:01.068+0000 7f73997fa640 1 -- 172.21.15.142:0/2189578587 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7f738c044de0 con 0x7f739c105b60 2024-09-06T14:20:02.774 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:01.229+0000 7f73a2405640 1 -- 172.21.15.142:0/2189578587 --> v2:172.21.15.142:6800/853729507 -- mgr_command(tid 0: {"prefix": "orch certmgr generate-certificates", "module_name": "dashboard", "target": ["mon-mgr", ""]}) -- 0x7f739c18f9b0 con 0x7f738803da50 2024-09-06T14:20:02.774 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:02.195+0000 7f73997fa640 1 -- 172.21.15.142:0/2189578587 <== mgr.14146 v2:172.21.15.142:6800/853729507 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+5136 (secure 0 0 0) 0x7f739c18f9b0 con 0x7f738803da50 2024-09-06T14:20:02.775 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:02.198+0000 7f73a2405640 1 -- 172.21.15.142:0/2189578587 >> v2:172.21.15.142:6800/853729507 conn(0x7f738803da50 msgr2=0x7f738803ff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:20:02.775 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:02.198+0000 7f73a2405640 1 --2- 172.21.15.142:0/2189578587 >> v2:172.21.15.142:6800/853729507 conn(0x7f738803da50 0x7f738803ff10 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f73840098c0 tx=0x7f7384006db0 comp rx=0 tx=0).stop 2024-09-06T14:20:02.775 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:02.198+0000 7f73a2405640 1 -- 172.21.15.142:0/2189578587 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f739c105b60 msgr2=0x7f739c190540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:20:02.775 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:02.198+0000 7f73a2405640 1 --2- 172.21.15.142:0/2189578587 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f739c105b60 0x7f739c190540 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f738c006f10 tx=0x7f738c032740 comp rx=0 tx=0).stop 2024-09-06T14:20:02.775 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:02.198+0000 7f73a2405640 1 -- 172.21.15.142:0/2189578587 shutdown_connections 2024-09-06T14:20:02.775 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:02.198+0000 7f73a2405640 1 --2- 172.21.15.142:0/2189578587 >> v2:172.21.15.142:6800/853729507 conn(0x7f738803da50 0x7f738803ff10 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:20:02.775 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:02.198+0000 7f73a2405640 1 --2- 172.21.15.142:0/2189578587 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f739c105b60 0x7f739c190540 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:20:02.775 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:02.198+0000 7f73a2405640 1 -- 172.21.15.142:0/2189578587 >> 172.21.15.142:0/2189578587 conn(0x7f739c100700 msgr2=0x7f739c104880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:20:02.775 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:02.198+0000 7f73a2405640 1 -- 172.21.15.142:0/2189578587 shutdown_connections 2024-09-06T14:20:02.775 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:02.198+0000 7f73a2405640 1 -- 172.21.15.142:0/2189578587 wait complete. 2024-09-06T14:20:03.038 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:02 smithi142 ceph-mon[29958]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "orch certmgr generate-certificates", "module_name": "dashboard", "target": ["mon-mgr", ""]}]: dispatch 2024-09-06T14:20:03.038 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:02 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:04.706 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:04 smithi142 ceph-mon[29958]: from='client.14158 -' entity='client.admin' cmd=[{"prefix": "dashboard set-ssl-certificate", "target": ["mon-mgr", ""]}]: dispatch 2024-09-06T14:20:04.707 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:04 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:04.840 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout SSL certificate updated 2024-09-06T14:20:04.841 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:03.239+0000 7fc6865ab640 1 Processor -- start 2024-09-06T14:20:04.841 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:03.239+0000 7fc6865ab640 1 -- start start 2024-09-06T14:20:04.841 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:03.239+0000 7fc6865ab640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc680106710 0x7fc680106b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:20:04.841 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:03.239+0000 7fc6865ab640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fc680107100 con 0x7fc680106710 2024-09-06T14:20:04.841 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:03.240+0000 7fc67ffff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc680106710 0x7fc680106b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:20:04.841 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:03.240+0000 7fc67ffff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc680106710 0x7fc680106b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:33536/0 (socket says 172.21.15.142:33536) 2024-09-06T14:20:04.841 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:03.240+0000 7fc67ffff640 1 -- 172.21.15.142:0/1991689566 learned_addr learned my addr 172.21.15.142:0/1991689566 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:20:04.841 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:03.240+0000 7fc67ffff640 1 -- 172.21.15.142:0/1991689566 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc680107280 con 0x7fc680106710 2024-09-06T14:20:04.841 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:03.241+0000 7fc67ffff640 1 --2- 172.21.15.142:0/1991689566 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc680106710 0x7fc680106b10 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc670009820 tx=0x7fc67002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd6f1f49df255d04 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:20:04.841 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:03.241+0000 7fc67effd640 1 -- 172.21.15.142:0/1991689566 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc670037070 con 0x7fc680106710 2024-09-06T14:20:04.841 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:03.241+0000 7fc67effd640 1 -- 172.21.15.142:0/1991689566 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7fc6700328c0 con 0x7fc680106710 2024-09-06T14:20:04.841 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:03.241+0000 7fc67effd640 1 -- 172.21.15.142:0/1991689566 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc670030640 con 0x7fc680106710 2024-09-06T14:20:04.841 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:03.241+0000 7fc6865ab640 1 -- 172.21.15.142:0/1991689566 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc680106710 msgr2=0x7fc680106b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:20:04.842 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:03.241+0000 7fc6865ab640 1 --2- 172.21.15.142:0/1991689566 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc680106710 0x7fc680106b10 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc670009820 tx=0x7fc67002a270 comp rx=0 tx=0).stop 2024-09-06T14:20:04.842 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:03.241+0000 7fc6865ab640 1 -- 172.21.15.142:0/1991689566 shutdown_connections 2024-09-06T14:20:04.842 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:03.241+0000 7fc6865ab640 1 --2- 172.21.15.142:0/1991689566 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc680106710 0x7fc680106b10 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:20:04.842 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:03.241+0000 7fc6865ab640 1 -- 172.21.15.142:0/1991689566 >> 172.21.15.142:0/1991689566 conn(0x7fc680102020 msgr2=0x7fc680104440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:20:04.842 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:03.241+0000 7fc6865ab640 1 -- 172.21.15.142:0/1991689566 shutdown_connections 2024-09-06T14:20:04.842 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:03.241+0000 7fc6865ab640 1 -- 172.21.15.142:0/1991689566 wait complete. 2024-09-06T14:20:04.842 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:03.242+0000 7fc6865ab640 1 Processor -- start 2024-09-06T14:20:04.842 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:03.242+0000 7fc6865ab640 1 -- start start 2024-09-06T14:20:04.842 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:03.242+0000 7fc6865ab640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc680106710 0x7fc68019a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:20:04.842 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:03.242+0000 7fc6865ab640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fc68010ab40 con 0x7fc680106710 2024-09-06T14:20:04.842 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:03.242+0000 7fc67ffff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc680106710 0x7fc68019a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:20:04.842 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:03.242+0000 7fc67ffff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc680106710 0x7fc68019a650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:33552/0 (socket says 172.21.15.142:33552) 2024-09-06T14:20:04.842 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:03.242+0000 7fc67ffff640 1 -- 172.21.15.142:0/3690891429 learned_addr learned my addr 172.21.15.142:0/3690891429 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:20:04.843 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:03.243+0000 7fc67ffff640 1 -- 172.21.15.142:0/3690891429 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc68019abb0 con 0x7fc680106710 2024-09-06T14:20:04.843 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:03.243+0000 7fc67ffff640 1 --2- 172.21.15.142:0/3690891429 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc680106710 0x7fc68019a650 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc670000c00 tx=0x7fc670030b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:20:04.843 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:03.243+0000 7fc67d7fa640 1 -- 172.21.15.142:0/3690891429 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc670037040 con 0x7fc680106710 2024-09-06T14:20:04.843 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:03.243+0000 7fc67d7fa640 1 -- 172.21.15.142:0/3690891429 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7fc670040070 con 0x7fc680106710 2024-09-06T14:20:04.843 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:03.243+0000 7fc67d7fa640 1 -- 172.21.15.142:0/3690891429 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc67003a390 con 0x7fc680106710 2024-09-06T14:20:04.843 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:03.243+0000 7fc6865ab640 1 -- 172.21.15.142:0/3690891429 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc68019ae40 con 0x7fc680106710 2024-09-06T14:20:04.843 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:03.243+0000 7fc6865ab640 1 -- 172.21.15.142:0/3690891429 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc68019b800 con 0x7fc680106710 2024-09-06T14:20:04.843 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:03.244+0000 7fc67d7fa640 1 -- 172.21.15.142:0/3690891429 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 12) ==== 50088+0+0 (secure 0 0 0) 0x7fc67003b430 con 0x7fc680106710 2024-09-06T14:20:04.843 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:03.244+0000 7fc6865ab640 1 -- 172.21.15.142:0/3690891429 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc680106b10 con 0x7fc680106710 2024-09-06T14:20:04.843 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:03.244+0000 7fc67d7fa640 1 --2- 172.21.15.142:0/3690891429 >> v2:172.21.15.142:6800/853729507 conn(0x7fc66803daa0 0x7fc66803ff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:20:04.843 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:03.244+0000 7fc67d7fa640 1 -- 172.21.15.142:0/3690891429 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 983+0+0 (secure 0 0 0) 0x7fc670071f10 con 0x7fc680106710 2024-09-06T14:20:04.843 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:03.244+0000 7fc67f7fe640 1 --2- 172.21.15.142:0/3690891429 >> v2:172.21.15.142:6800/853729507 conn(0x7fc66803daa0 0x7fc66803ff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:20:04.843 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:03.245+0000 7fc67f7fe640 1 --2- 172.21.15.142:0/3690891429 >> v2:172.21.15.142:6800/853729507 conn(0x7fc66803daa0 0x7fc66803ff60 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fc6740098c0 tx=0x7fc674006db0 comp rx=0 tx=0).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:20:04.843 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:03.249+0000 7fc67d7fa640 1 -- 172.21.15.142:0/3690891429 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7fc670036da0 con 0x7fc680106710 2024-09-06T14:20:04.844 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:03.431+0000 7fc6865ab640 1 -- 172.21.15.142:0/3690891429 --> v2:172.21.15.142:6800/853729507 -- mgr_command(tid 0: {"prefix": "dashboard set-ssl-certificate", "target": ["mon-mgr", ""]}) -- 0x7fc6801031f0 con 0x7fc66803daa0 2024-09-06T14:20:04.844 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:03.434+0000 7fc67d7fa640 1 -- 172.21.15.142:0/3690891429 <== mgr.14146 v2:172.21.15.142:6800/853729507 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fc6801031f0 con 0x7fc66803daa0 2024-09-06T14:20:04.844 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:03.436+0000 7fc6865ab640 1 -- 172.21.15.142:0/3690891429 >> v2:172.21.15.142:6800/853729507 conn(0x7fc66803daa0 msgr2=0x7fc66803ff60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:20:04.844 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:03.436+0000 7fc6865ab640 1 --2- 172.21.15.142:0/3690891429 >> v2:172.21.15.142:6800/853729507 conn(0x7fc66803daa0 0x7fc66803ff60 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fc6740098c0 tx=0x7fc674006db0 comp rx=0 tx=0).stop 2024-09-06T14:20:04.844 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:03.437+0000 7fc6865ab640 1 -- 172.21.15.142:0/3690891429 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc680106710 msgr2=0x7fc68019a650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:20:04.844 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:03.437+0000 7fc6865ab640 1 --2- 172.21.15.142:0/3690891429 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc680106710 0x7fc68019a650 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc670000c00 tx=0x7fc670030b70 comp rx=0 tx=0).stop 2024-09-06T14:20:04.844 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:03.437+0000 7fc6865ab640 1 -- 172.21.15.142:0/3690891429 shutdown_connections 2024-09-06T14:20:04.844 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:03.437+0000 7fc6865ab640 1 --2- 172.21.15.142:0/3690891429 >> v2:172.21.15.142:6800/853729507 conn(0x7fc66803daa0 0x7fc66803ff60 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:20:04.844 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:03.437+0000 7fc6865ab640 1 --2- 172.21.15.142:0/3690891429 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc680106710 0x7fc68019a650 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:20:04.844 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:03.437+0000 7fc6865ab640 1 -- 172.21.15.142:0/3690891429 >> 172.21.15.142:0/3690891429 conn(0x7fc680102020 msgr2=0x7fc680102ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:20:04.844 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:03.437+0000 7fc6865ab640 1 -- 172.21.15.142:0/3690891429 shutdown_connections 2024-09-06T14:20:04.844 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:03.437+0000 7fc6865ab640 1 -- 172.21.15.142:0/3690891429 wait complete. 2024-09-06T14:20:06.181 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout SSL certificate key updated 2024-09-06T14:20:06.181 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:05.412+0000 7f9653273640 1 Processor -- start 2024-09-06T14:20:06.181 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:05.413+0000 7f9653273640 1 -- start start 2024-09-06T14:20:06.181 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:05.413+0000 7f9653273640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f964c1055b0 0x7f964c1059b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:20:06.181 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:05.413+0000 7f9653273640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f964c105fa0 con 0x7f964c1055b0 2024-09-06T14:20:06.182 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:05.413+0000 7f965100f640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f964c1055b0 0x7f964c1059b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:20:06.182 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:05.413+0000 7f965100f640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f964c1055b0 0x7f964c1059b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:33562/0 (socket says 172.21.15.142:33562) 2024-09-06T14:20:06.182 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:05.413+0000 7f965100f640 1 -- 172.21.15.142:0/1505059537 learned_addr learned my addr 172.21.15.142:0/1505059537 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:20:06.182 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:05.413+0000 7f965100f640 1 -- 172.21.15.142:0/1505059537 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f964c106120 con 0x7f964c1055b0 2024-09-06T14:20:06.182 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:05.414+0000 7f965100f640 1 --2- 172.21.15.142:0/1505059537 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f964c1055b0 0x7f964c1059b0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f9648009820 tx=0x7f964802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9e311154d636a85 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:20:06.182 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:05.414+0000 7f9643fff640 1 -- 172.21.15.142:0/1505059537 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9648037070 con 0x7f964c1055b0 2024-09-06T14:20:06.183 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:05.414+0000 7f9643fff640 1 -- 172.21.15.142:0/1505059537 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7f96480328c0 con 0x7f964c1055b0 2024-09-06T14:20:06.183 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:05.414+0000 7f9643fff640 1 -- 172.21.15.142:0/1505059537 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9648030640 con 0x7f964c1055b0 2024-09-06T14:20:06.183 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:05.415+0000 7f9653273640 1 -- 172.21.15.142:0/1505059537 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f964c1055b0 msgr2=0x7f964c1059b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:20:06.183 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:05.415+0000 7f9653273640 1 --2- 172.21.15.142:0/1505059537 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f964c1055b0 0x7f964c1059b0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f9648009820 tx=0x7f964802a270 comp rx=0 tx=0).stop 2024-09-06T14:20:06.183 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:05.415+0000 7f9653273640 1 -- 172.21.15.142:0/1505059537 shutdown_connections 2024-09-06T14:20:06.183 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:05.415+0000 7f9653273640 1 --2- 172.21.15.142:0/1505059537 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f964c1055b0 0x7f964c1059b0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:20:06.183 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:05.415+0000 7f9653273640 1 -- 172.21.15.142:0/1505059537 >> 172.21.15.142:0/1505059537 conn(0x7f964c100f40 msgr2=0x7f964c103360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:20:06.183 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:05.415+0000 7f9653273640 1 -- 172.21.15.142:0/1505059537 shutdown_connections 2024-09-06T14:20:06.184 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:05.415+0000 7f9653273640 1 -- 172.21.15.142:0/1505059537 wait complete. 2024-09-06T14:20:06.184 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:05.415+0000 7f9653273640 1 Processor -- start 2024-09-06T14:20:06.184 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:05.415+0000 7f9653273640 1 -- start start 2024-09-06T14:20:06.184 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:05.415+0000 7f9653273640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f964c1055b0 0x7f964c079850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:20:06.184 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:05.415+0000 7f9653273640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f964c109a30 con 0x7f964c1055b0 2024-09-06T14:20:06.184 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:05.416+0000 7f965100f640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f964c1055b0 0x7f964c079850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:20:06.184 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:05.416+0000 7f965100f640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f964c1055b0 0x7f964c079850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:33564/0 (socket says 172.21.15.142:33564) 2024-09-06T14:20:06.184 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:05.416+0000 7f965100f640 1 -- 172.21.15.142:0/1716092894 learned_addr learned my addr 172.21.15.142:0/1716092894 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:20:06.185 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:05.416+0000 7f965100f640 1 -- 172.21.15.142:0/1716092894 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f964c079db0 con 0x7f964c1055b0 2024-09-06T14:20:06.185 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:05.416+0000 7f965100f640 1 --2- 172.21.15.142:0/1716092894 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f964c1055b0 0x7f964c079850 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f964802a830 tx=0x7f9648030bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:20:06.185 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:05.416+0000 7f96427fc640 1 -- 172.21.15.142:0/1716092894 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9648037040 con 0x7f964c1055b0 2024-09-06T14:20:06.185 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:05.416+0000 7f96427fc640 1 -- 172.21.15.142:0/1716092894 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7f9648040070 con 0x7f964c1055b0 2024-09-06T14:20:06.185 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:05.416+0000 7f9653273640 1 -- 172.21.15.142:0/1716092894 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f964c19d390 con 0x7f964c1055b0 2024-09-06T14:20:06.185 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:05.416+0000 7f96427fc640 1 -- 172.21.15.142:0/1716092894 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f964803be40 con 0x7f964c1055b0 2024-09-06T14:20:06.185 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:05.416+0000 7f9653273640 1 -- 172.21.15.142:0/1716092894 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f964c19d860 con 0x7f964c1055b0 2024-09-06T14:20:06.185 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:05.417+0000 7f96427fc640 1 -- 172.21.15.142:0/1716092894 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 12) ==== 50088+0+0 (secure 0 0 0) 0x7f964803b430 con 0x7f964c1055b0 2024-09-06T14:20:06.185 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:05.417+0000 7f9653273640 1 -- 172.21.15.142:0/1716092894 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f964c1059b0 con 0x7f964c1055b0 2024-09-06T14:20:06.186 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:05.418+0000 7f96427fc640 1 --2- 172.21.15.142:0/1716092894 >> v2:172.21.15.142:6800/853729507 conn(0x7f963803da50 0x7f963803ff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:20:06.186 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:05.418+0000 7f96427fc640 1 -- 172.21.15.142:0/1716092894 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 983+0+0 (secure 0 0 0) 0x7f9648071ef0 con 0x7f964c1055b0 2024-09-06T14:20:06.186 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:05.418+0000 7f965080e640 1 --2- 172.21.15.142:0/1716092894 >> v2:172.21.15.142:6800/853729507 conn(0x7f963803da50 0x7f963803ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:20:06.186 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:05.418+0000 7f965080e640 1 --2- 172.21.15.142:0/1716092894 >> v2:172.21.15.142:6800/853729507 conn(0x7f963803da50 0x7f963803ff10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f963c0098c0 tx=0x7f963c006db0 comp rx=0 tx=0).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:20:06.186 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:05.422+0000 7f96427fc640 1 -- 172.21.15.142:0/1716092894 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7f964802e200 con 0x7f964c1055b0 2024-09-06T14:20:06.186 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:05.587+0000 7f9653273640 1 -- 172.21.15.142:0/1716092894 --> v2:172.21.15.142:6800/853729507 -- mgr_command(tid 0: {"prefix": "dashboard set-ssl-certificate-key", "target": ["mon-mgr", ""]}) -- 0x7f964c102140 con 0x7f963803da50 2024-09-06T14:20:06.186 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:05.590+0000 7f96427fc640 1 -- 172.21.15.142:0/1716092894 <== mgr.14146 v2:172.21.15.142:6800/853729507 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7f964c102140 con 0x7f963803da50 2024-09-06T14:20:06.186 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:05.592+0000 7f9653273640 1 -- 172.21.15.142:0/1716092894 >> v2:172.21.15.142:6800/853729507 conn(0x7f963803da50 msgr2=0x7f963803ff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:20:06.187 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:05.592+0000 7f9653273640 1 --2- 172.21.15.142:0/1716092894 >> v2:172.21.15.142:6800/853729507 conn(0x7f963803da50 0x7f963803ff10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f963c0098c0 tx=0x7f963c006db0 comp rx=0 tx=0).stop 2024-09-06T14:20:06.187 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:05.592+0000 7f9653273640 1 -- 172.21.15.142:0/1716092894 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f964c1055b0 msgr2=0x7f964c079850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:20:06.187 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:05.592+0000 7f9653273640 1 --2- 172.21.15.142:0/1716092894 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f964c1055b0 0x7f964c079850 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f964802a830 tx=0x7f9648030bc0 comp rx=0 tx=0).stop 2024-09-06T14:20:06.187 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:05.592+0000 7f9653273640 1 -- 172.21.15.142:0/1716092894 shutdown_connections 2024-09-06T14:20:06.187 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:05.592+0000 7f9653273640 1 --2- 172.21.15.142:0/1716092894 >> v2:172.21.15.142:6800/853729507 conn(0x7f963803da50 0x7f963803ff10 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:20:06.187 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:05.592+0000 7f9653273640 1 --2- 172.21.15.142:0/1716092894 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f964c1055b0 0x7f964c079850 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:20:06.187 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:05.592+0000 7f9653273640 1 -- 172.21.15.142:0/1716092894 >> 172.21.15.142:0/1716092894 conn(0x7f964c100f40 msgr2=0x7f964c101a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:20:06.187 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:05.592+0000 7f9653273640 1 -- 172.21.15.142:0/1716092894 shutdown_connections 2024-09-06T14:20:06.187 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:05.592+0000 7f9653273640 1 -- 172.21.15.142:0/1716092894 wait complete. 2024-09-06T14:20:06.188 INFO:teuthology.orchestra.run.smithi142.stdout:Creating initial admin user... 2024-09-06T14:20:06.744 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:06 smithi142 ceph-mon[29958]: from='client.14160 -' entity='client.admin' cmd=[{"prefix": "dashboard set-ssl-certificate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-06T14:20:06.744 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:06 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:07.627 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$jMVNskIiME08RzQNu5t16eubMFaJXbLs6t3CsODshr4AMp05zcetW", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1725632406, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-09-06T14:20:07.627 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:06.576+0000 7fdb860e6640 1 Processor -- start 2024-09-06T14:20:07.627 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:06.576+0000 7fdb860e6640 1 -- start start 2024-09-06T14:20:07.627 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:06.576+0000 7fdb860e6640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdb801055f0 0x7fdb801059f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:20:07.627 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:06.576+0000 7fdb860e6640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fdb80105fe0 con 0x7fdb801055f0 2024-09-06T14:20:07.627 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:06.576+0000 7fdb7f7fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdb801055f0 0x7fdb801059f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:20:07.628 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:06.576+0000 7fdb7f7fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdb801055f0 0x7fdb801059f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:33576/0 (socket says 172.21.15.142:33576) 2024-09-06T14:20:07.628 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:06.576+0000 7fdb7f7fe640 1 -- 172.21.15.142:0/4254139650 learned_addr learned my addr 172.21.15.142:0/4254139650 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:20:07.628 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:06.576+0000 7fdb7f7fe640 1 -- 172.21.15.142:0/4254139650 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb80106160 con 0x7fdb801055f0 2024-09-06T14:20:07.628 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:06.577+0000 7fdb7f7fe640 1 --2- 172.21.15.142:0/4254139650 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdb801055f0 0x7fdb801059f0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fdb70009820 tx=0x7fdb7002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=436a99870961c1c6 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:20:07.628 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:06.577+0000 7fdb7e7fc640 1 -- 172.21.15.142:0/4254139650 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdb70037070 con 0x7fdb801055f0 2024-09-06T14:20:07.628 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:06.577+0000 7fdb7e7fc640 1 -- 172.21.15.142:0/4254139650 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7fdb700328c0 con 0x7fdb801055f0 2024-09-06T14:20:07.628 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:06.577+0000 7fdb7e7fc640 1 -- 172.21.15.142:0/4254139650 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdb70030640 con 0x7fdb801055f0 2024-09-06T14:20:07.628 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:06.578+0000 7fdb860e6640 1 -- 172.21.15.142:0/4254139650 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdb801055f0 msgr2=0x7fdb801059f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:20:07.628 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:06.578+0000 7fdb860e6640 1 --2- 172.21.15.142:0/4254139650 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdb801055f0 0x7fdb801059f0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fdb70009820 tx=0x7fdb7002a270 comp rx=0 tx=0).stop 2024-09-06T14:20:07.628 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:06.578+0000 7fdb860e6640 1 -- 172.21.15.142:0/4254139650 shutdown_connections 2024-09-06T14:20:07.628 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:06.578+0000 7fdb860e6640 1 --2- 172.21.15.142:0/4254139650 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdb801055f0 0x7fdb801059f0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:20:07.628 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:06.578+0000 7fdb860e6640 1 -- 172.21.15.142:0/4254139650 >> 172.21.15.142:0/4254139650 conn(0x7fdb80100ee0 msgr2=0x7fdb80103300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:20:07.628 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:06.578+0000 7fdb860e6640 1 -- 172.21.15.142:0/4254139650 shutdown_connections 2024-09-06T14:20:07.629 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:06.578+0000 7fdb860e6640 1 -- 172.21.15.142:0/4254139650 wait complete. 2024-09-06T14:20:07.629 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:06.578+0000 7fdb860e6640 1 Processor -- start 2024-09-06T14:20:07.629 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:06.579+0000 7fdb860e6640 1 -- start start 2024-09-06T14:20:07.629 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:06.579+0000 7fdb860e6640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdb801055f0 0x7fdb8007a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:20:07.629 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:06.579+0000 7fdb860e6640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fdb80109a70 con 0x7fdb801055f0 2024-09-06T14:20:07.629 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:06.579+0000 7fdb7f7fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdb801055f0 0x7fdb8007a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:20:07.629 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:06.579+0000 7fdb7f7fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdb801055f0 0x7fdb8007a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:33580/0 (socket says 172.21.15.142:33580) 2024-09-06T14:20:07.629 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:06.579+0000 7fdb7f7fe640 1 -- 172.21.15.142:0/76521004 learned_addr learned my addr 172.21.15.142:0/76521004 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:20:07.629 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:06.579+0000 7fdb7f7fe640 1 -- 172.21.15.142:0/76521004 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb8007a640 con 0x7fdb801055f0 2024-09-06T14:20:07.629 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:06.579+0000 7fdb7f7fe640 1 --2- 172.21.15.142:0/76521004 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdb801055f0 0x7fdb8007a0e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fdb7002a830 tx=0x7fdb70030bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:20:07.629 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:06.579+0000 7fdb7cff9640 1 -- 172.21.15.142:0/76521004 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdb70037040 con 0x7fdb801055f0 2024-09-06T14:20:07.629 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:06.580+0000 7fdb860e6640 1 -- 172.21.15.142:0/76521004 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb8019d360 con 0x7fdb801055f0 2024-09-06T14:20:07.629 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:06.580+0000 7fdb7cff9640 1 -- 172.21.15.142:0/76521004 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7fdb70040070 con 0x7fdb801055f0 2024-09-06T14:20:07.629 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:06.580+0000 7fdb7cff9640 1 -- 172.21.15.142:0/76521004 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdb7003be40 con 0x7fdb801055f0 2024-09-06T14:20:07.630 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:06.580+0000 7fdb860e6640 1 -- 172.21.15.142:0/76521004 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb8019d830 con 0x7fdb801055f0 2024-09-06T14:20:07.630 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:06.580+0000 7fdb7cff9640 1 -- 172.21.15.142:0/76521004 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 12) ==== 50088+0+0 (secure 0 0 0) 0x7fdb7003a4f0 con 0x7fdb801055f0 2024-09-06T14:20:07.630 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:06.581+0000 7fdb860e6640 1 -- 172.21.15.142:0/76521004 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb80109bf0 con 0x7fdb801055f0 2024-09-06T14:20:07.630 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:06.581+0000 7fdb7cff9640 1 --2- 172.21.15.142:0/76521004 >> v2:172.21.15.142:6800/853729507 conn(0x7fdb6803da50 0x7fdb6803ff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:20:07.630 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:06.581+0000 7fdb7cff9640 1 -- 172.21.15.142:0/76521004 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 983+0+0 (secure 0 0 0) 0x7fdb70071f10 con 0x7fdb801055f0 2024-09-06T14:20:07.630 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:06.581+0000 7fdb7effd640 1 --2- 172.21.15.142:0/76521004 >> v2:172.21.15.142:6800/853729507 conn(0x7fdb6803da50 0x7fdb6803ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:20:07.630 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:06.581+0000 7fdb7effd640 1 --2- 172.21.15.142:0/76521004 >> v2:172.21.15.142:6800/853729507 conn(0x7fdb6803da50 0x7fdb6803ff10 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fdb740098c0 tx=0x7fdb74006db0 comp rx=0 tx=0).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:20:07.630 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:06.585+0000 7fdb7cff9640 1 -- 172.21.15.142:0/76521004 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7fdb70030d50 con 0x7fdb801055f0 2024-09-06T14:20:07.630 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:06.748+0000 7fdb860e6640 1 -- 172.21.15.142:0/76521004 --> v2:172.21.15.142:6800/853729507 -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7fdb8019dba0 con 0x7fdb6803da50 2024-09-06T14:20:07.630 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:06.970+0000 7fdb7cff9640 1 -- 172.21.15.142:0/76521004 <== mgr.14146 v2:172.21.15.142:6800/853729507 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7fdb8019dba0 con 0x7fdb6803da50 2024-09-06T14:20:07.630 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:06.972+0000 7fdb860e6640 1 -- 172.21.15.142:0/76521004 >> v2:172.21.15.142:6800/853729507 conn(0x7fdb6803da50 msgr2=0x7fdb6803ff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:20:07.630 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:06.972+0000 7fdb860e6640 1 --2- 172.21.15.142:0/76521004 >> v2:172.21.15.142:6800/853729507 conn(0x7fdb6803da50 0x7fdb6803ff10 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fdb740098c0 tx=0x7fdb74006db0 comp rx=0 tx=0).stop 2024-09-06T14:20:07.631 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:06.972+0000 7fdb860e6640 1 -- 172.21.15.142:0/76521004 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdb801055f0 msgr2=0x7fdb8007a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:20:07.631 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:06.972+0000 7fdb860e6640 1 --2- 172.21.15.142:0/76521004 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdb801055f0 0x7fdb8007a0e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fdb7002a830 tx=0x7fdb70030bc0 comp rx=0 tx=0).stop 2024-09-06T14:20:07.631 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:06.972+0000 7fdb860e6640 1 -- 172.21.15.142:0/76521004 shutdown_connections 2024-09-06T14:20:07.631 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:06.972+0000 7fdb860e6640 1 --2- 172.21.15.142:0/76521004 >> v2:172.21.15.142:6800/853729507 conn(0x7fdb6803da50 0x7fdb6803ff10 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:20:07.631 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:06.972+0000 7fdb860e6640 1 --2- 172.21.15.142:0/76521004 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdb801055f0 0x7fdb8007a0e0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:20:07.631 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:06.972+0000 7fdb860e6640 1 -- 172.21.15.142:0/76521004 >> 172.21.15.142:0/76521004 conn(0x7fdb80100ee0 msgr2=0x7fdb80101a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:20:07.631 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:06.972+0000 7fdb860e6640 1 -- 172.21.15.142:0/76521004 shutdown_connections 2024-09-06T14:20:07.631 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:06.972+0000 7fdb860e6640 1 -- 172.21.15.142:0/76521004 wait complete. 2024-09-06T14:20:07.631 INFO:teuthology.orchestra.run.smithi142.stdout:Fetching dashboard port number... 2024-09-06T14:20:07.850 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:07 smithi142 ceph-mon[29958]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-06T14:20:07.850 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:07 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:08.794 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stdout 8443 2024-09-06T14:20:08.794 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:08.060+0000 7fc089f60640 1 Processor -- start 2024-09-06T14:20:08.794 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:08.061+0000 7fc089f60640 1 -- start start 2024-09-06T14:20:08.794 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:08.061+0000 7fc089f60640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc0841055b0 0x7fc0841059b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:20:08.794 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:08.061+0000 7fc089f60640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fc084105fa0 con 0x7fc0841055b0 2024-09-06T14:20:08.794 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:08.061+0000 7fc0837fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc0841055b0 0x7fc0841059b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:20:08.794 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:08.061+0000 7fc0837fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc0841055b0 0x7fc0841059b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:41152/0 (socket says 172.21.15.142:41152) 2024-09-06T14:20:08.794 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:08.061+0000 7fc0837fe640 1 -- 172.21.15.142:0/1179080221 learned_addr learned my addr 172.21.15.142:0/1179080221 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:20:08.794 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:08.061+0000 7fc0837fe640 1 -- 172.21.15.142:0/1179080221 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc084106120 con 0x7fc0841055b0 2024-09-06T14:20:08.795 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:08.062+0000 7fc0837fe640 1 --2- 172.21.15.142:0/1179080221 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc0841055b0 0x7fc0841059b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc074009820 tx=0x7fc07402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c651569472d46057 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:20:08.795 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:08.062+0000 7fc0827fc640 1 -- 172.21.15.142:0/1179080221 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc074037070 con 0x7fc0841055b0 2024-09-06T14:20:08.795 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:08.062+0000 7fc0827fc640 1 -- 172.21.15.142:0/1179080221 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7fc0740328c0 con 0x7fc0841055b0 2024-09-06T14:20:08.795 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:08.062+0000 7fc0827fc640 1 -- 172.21.15.142:0/1179080221 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc074030640 con 0x7fc0841055b0 2024-09-06T14:20:08.795 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:08.063+0000 7fc089f60640 1 -- 172.21.15.142:0/1179080221 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc0841055b0 msgr2=0x7fc0841059b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:20:08.795 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:08.063+0000 7fc089f60640 1 --2- 172.21.15.142:0/1179080221 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc0841055b0 0x7fc0841059b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc074009820 tx=0x7fc07402a270 comp rx=0 tx=0).stop 2024-09-06T14:20:08.795 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:08.063+0000 7fc089f60640 1 -- 172.21.15.142:0/1179080221 shutdown_connections 2024-09-06T14:20:08.795 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:08.063+0000 7fc089f60640 1 --2- 172.21.15.142:0/1179080221 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc0841055b0 0x7fc0841059b0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:20:08.795 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:08.063+0000 7fc089f60640 1 -- 172.21.15.142:0/1179080221 >> 172.21.15.142:0/1179080221 conn(0x7fc084100f40 msgr2=0x7fc084103360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:20:08.795 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:08.063+0000 7fc089f60640 1 -- 172.21.15.142:0/1179080221 shutdown_connections 2024-09-06T14:20:08.795 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:08.063+0000 7fc089f60640 1 -- 172.21.15.142:0/1179080221 wait complete. 2024-09-06T14:20:08.795 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:08.063+0000 7fc089f60640 1 Processor -- start 2024-09-06T14:20:08.795 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:08.064+0000 7fc089f60640 1 -- start start 2024-09-06T14:20:08.795 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:08.064+0000 7fc089f60640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc0841055b0 0x7fc08419a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:20:08.796 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:08.064+0000 7fc089f60640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fc084109a30 con 0x7fc0841055b0 2024-09-06T14:20:08.796 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:08.064+0000 7fc0837fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc0841055b0 0x7fc08419a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:20:08.796 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:08.064+0000 7fc0837fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc0841055b0 0x7fc08419a680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:41154/0 (socket says 172.21.15.142:41154) 2024-09-06T14:20:08.796 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:08.064+0000 7fc0837fe640 1 -- 172.21.15.142:0/3755656954 learned_addr learned my addr 172.21.15.142:0/3755656954 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:20:08.796 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:08.064+0000 7fc0837fe640 1 -- 172.21.15.142:0/3755656954 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc08419abe0 con 0x7fc0841055b0 2024-09-06T14:20:08.796 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:08.064+0000 7fc0837fe640 1 --2- 172.21.15.142:0/3755656954 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc0841055b0 0x7fc08419a680 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc07402a830 tx=0x7fc074030bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:20:08.796 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:08.064+0000 7fc080ff9640 1 -- 172.21.15.142:0/3755656954 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc074037040 con 0x7fc0841055b0 2024-09-06T14:20:08.796 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:08.064+0000 7fc080ff9640 1 -- 172.21.15.142:0/3755656954 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7fc074040070 con 0x7fc0841055b0 2024-09-06T14:20:08.796 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:08.064+0000 7fc089f60640 1 -- 172.21.15.142:0/3755656954 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc08419ae70 con 0x7fc0841055b0 2024-09-06T14:20:08.796 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:08.065+0000 7fc080ff9640 1 -- 172.21.15.142:0/3755656954 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc07403be40 con 0x7fc0841055b0 2024-09-06T14:20:08.796 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:08.065+0000 7fc089f60640 1 -- 172.21.15.142:0/3755656954 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc08419b830 con 0x7fc0841055b0 2024-09-06T14:20:08.796 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:08.065+0000 7fc080ff9640 1 -- 172.21.15.142:0/3755656954 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 12) ==== 50088+0+0 (secure 0 0 0) 0x7fc07403a500 con 0x7fc0841055b0 2024-09-06T14:20:08.796 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:08.066+0000 7fc080ff9640 1 --2- 172.21.15.142:0/3755656954 >> v2:172.21.15.142:6800/853729507 conn(0x7fc07003daa0 0x7fc07003ff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:20:08.797 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:08.066+0000 7fc080ff9640 1 -- 172.21.15.142:0/3755656954 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 983+0+0 (secure 0 0 0) 0x7fc074071ec0 con 0x7fc0841055b0 2024-09-06T14:20:08.797 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:08.066+0000 7fc089f60640 1 -- 172.21.15.142:0/3755656954 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0841059b0 con 0x7fc0841055b0 2024-09-06T14:20:08.797 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:08.066+0000 7fc082ffd640 1 --2- 172.21.15.142:0/3755656954 >> v2:172.21.15.142:6800/853729507 conn(0x7fc07003daa0 0x7fc07003ff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:20:08.797 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:08.066+0000 7fc082ffd640 1 --2- 172.21.15.142:0/3755656954 >> v2:172.21.15.142:6800/853729507 conn(0x7fc07003daa0 0x7fc07003ff60 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fc06c0098c0 tx=0x7fc06c006db0 comp rx=0 tx=0).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:20:08.797 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:08.070+0000 7fc080ff9640 1 -- 172.21.15.142:0/3755656954 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7fc074030d50 con 0x7fc0841055b0 2024-09-06T14:20:08.797 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:08.222+0000 7fc089f60640 1 -- 172.21.15.142:0/3755656954 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7fc08419bd40 con 0x7fc0841055b0 2024-09-06T14:20:08.797 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:08.222+0000 7fc080ff9640 1 -- 172.21.15.142:0/3755656954 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7fc07402ed40 con 0x7fc0841055b0 2024-09-06T14:20:08.797 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:08.225+0000 7fc089f60640 1 -- 172.21.15.142:0/3755656954 >> v2:172.21.15.142:6800/853729507 conn(0x7fc07003daa0 msgr2=0x7fc07003ff60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:20:08.797 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:08.225+0000 7fc089f60640 1 --2- 172.21.15.142:0/3755656954 >> v2:172.21.15.142:6800/853729507 conn(0x7fc07003daa0 0x7fc07003ff60 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fc06c0098c0 tx=0x7fc06c006db0 comp rx=0 tx=0).stop 2024-09-06T14:20:08.797 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:08.225+0000 7fc089f60640 1 -- 172.21.15.142:0/3755656954 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc0841055b0 msgr2=0x7fc08419a680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:20:08.797 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:08.225+0000 7fc089f60640 1 --2- 172.21.15.142:0/3755656954 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc0841055b0 0x7fc08419a680 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc07402a830 tx=0x7fc074030bc0 comp rx=0 tx=0).stop 2024-09-06T14:20:08.797 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:08.225+0000 7fc089f60640 1 -- 172.21.15.142:0/3755656954 shutdown_connections 2024-09-06T14:20:08.798 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:08.225+0000 7fc089f60640 1 --2- 172.21.15.142:0/3755656954 >> v2:172.21.15.142:6800/853729507 conn(0x7fc07003daa0 0x7fc07003ff60 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:20:08.798 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:08.225+0000 7fc089f60640 1 --2- 172.21.15.142:0/3755656954 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc0841055b0 0x7fc08419a680 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:20:08.798 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:08.225+0000 7fc089f60640 1 -- 172.21.15.142:0/3755656954 >> 172.21.15.142:0/3755656954 conn(0x7fc084100f40 msgr2=0x7fc0840790a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:20:08.798 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:08.225+0000 7fc089f60640 1 -- 172.21.15.142:0/3755656954 shutdown_connections 2024-09-06T14:20:08.798 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:08.225+0000 7fc089f60640 1 -- 172.21.15.142:0/3755656954 wait complete. 2024-09-06T14:20:08.798 INFO:teuthology.orchestra.run.smithi142.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-06T14:20:08.798 INFO:teuthology.orchestra.run.smithi142.stdout:systemctl: stdout disabled 2024-09-06T14:20:08.805 INFO:teuthology.orchestra.run.smithi142.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-06T14:20:08.806 INFO:teuthology.orchestra.run.smithi142.stdout:systemctl: stdout inactive 2024-09-06T14:20:08.806 INFO:teuthology.orchestra.run.smithi142.stdout:firewalld.service is not enabled 2024-09-06T14:20:08.806 INFO:teuthology.orchestra.run.smithi142.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-09-06T14:20:08.807 INFO:teuthology.orchestra.run.smithi142.stdout:Ceph Dashboard is now available at: 2024-09-06T14:20:08.807 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:20:08.807 INFO:teuthology.orchestra.run.smithi142.stdout: URL: https://smithi142:8443/ 2024-09-06T14:20:08.807 INFO:teuthology.orchestra.run.smithi142.stdout: User: admin 2024-09-06T14:20:08.807 INFO:teuthology.orchestra.run.smithi142.stdout: Password: sd2999fro9 2024-09-06T14:20:08.808 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:20:08.808 INFO:teuthology.orchestra.run.smithi142.stdout:Saving cluster configuration to /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/config directory 2024-09-06T14:20:08.928 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:08 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/3755656954' entity='client.admin' cmd={"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} : dispatch 2024-09-06T14:20:10.139 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:09.246+0000 7f1dc6dee640 1 Processor -- start 2024-09-06T14:20:10.139 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:09.246+0000 7f1dc6dee640 1 -- start start 2024-09-06T14:20:10.140 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:09.246+0000 7f1dc6dee640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1dc01055b0 0x7f1dc01059b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:20:10.140 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:09.246+0000 7f1dc6dee640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f1dc0105fa0 con 0x7f1dc01055b0 2024-09-06T14:20:10.140 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:09.246+0000 7f1dc4b8a640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1dc01055b0 0x7f1dc01059b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:20:10.140 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:09.247+0000 7f1dc4b8a640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1dc01055b0 0x7f1dc01059b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:41170/0 (socket says 172.21.15.142:41170) 2024-09-06T14:20:10.140 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:09.247+0000 7f1dc4b8a640 1 -- 172.21.15.142:0/78110353 learned_addr learned my addr 172.21.15.142:0/78110353 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:20:10.140 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:09.247+0000 7f1dc4b8a640 1 -- 172.21.15.142:0/78110353 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1dc0106120 con 0x7f1dc01055b0 2024-09-06T14:20:10.140 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:09.247+0000 7f1dc4b8a640 1 --2- 172.21.15.142:0/78110353 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1dc01055b0 0x7f1dc01059b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f1db4009820 tx=0x7f1db402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f89380cd5593cc2 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:20:10.140 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:09.248+0000 7f1dbf7fe640 1 -- 172.21.15.142:0/78110353 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1db4037070 con 0x7f1dc01055b0 2024-09-06T14:20:10.140 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:09.248+0000 7f1dbf7fe640 1 -- 172.21.15.142:0/78110353 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7f1db40328c0 con 0x7f1dc01055b0 2024-09-06T14:20:10.140 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:09.248+0000 7f1dbf7fe640 1 -- 172.21.15.142:0/78110353 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1db4030640 con 0x7f1dc01055b0 2024-09-06T14:20:10.140 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:09.248+0000 7f1dc6dee640 1 -- 172.21.15.142:0/78110353 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1dc01055b0 msgr2=0x7f1dc01059b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:20:10.140 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:09.248+0000 7f1dc6dee640 1 --2- 172.21.15.142:0/78110353 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1dc01055b0 0x7f1dc01059b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f1db4009820 tx=0x7f1db402a270 comp rx=0 tx=0).stop 2024-09-06T14:20:10.141 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:09.248+0000 7f1dc6dee640 1 -- 172.21.15.142:0/78110353 shutdown_connections 2024-09-06T14:20:10.141 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:09.248+0000 7f1dc6dee640 1 --2- 172.21.15.142:0/78110353 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1dc01055b0 0x7f1dc01059b0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:20:10.141 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:09.248+0000 7f1dc6dee640 1 -- 172.21.15.142:0/78110353 >> 172.21.15.142:0/78110353 conn(0x7f1dc0100f40 msgr2=0x7f1dc0103360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:20:10.141 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:09.248+0000 7f1dc6dee640 1 -- 172.21.15.142:0/78110353 shutdown_connections 2024-09-06T14:20:10.141 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:09.248+0000 7f1dc6dee640 1 -- 172.21.15.142:0/78110353 wait complete. 2024-09-06T14:20:10.141 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:09.249+0000 7f1dc6dee640 1 Processor -- start 2024-09-06T14:20:10.141 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:09.249+0000 7f1dc6dee640 1 -- start start 2024-09-06T14:20:10.141 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:09.249+0000 7f1dc6dee640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1dc01055b0 0x7f1dc019a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:20:10.141 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:09.249+0000 7f1dc6dee640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f1dc0109a30 con 0x7f1dc01055b0 2024-09-06T14:20:10.141 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:09.249+0000 7f1dc4b8a640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1dc01055b0 0x7f1dc019a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:20:10.141 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:09.249+0000 7f1dc4b8a640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1dc01055b0 0x7f1dc019a680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:41174/0 (socket says 172.21.15.142:41174) 2024-09-06T14:20:10.141 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:09.249+0000 7f1dc4b8a640 1 -- 172.21.15.142:0/1433029088 learned_addr learned my addr 172.21.15.142:0/1433029088 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:20:10.141 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:09.249+0000 7f1dc4b8a640 1 -- 172.21.15.142:0/1433029088 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1dc019abe0 con 0x7f1dc01055b0 2024-09-06T14:20:10.142 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:09.250+0000 7f1dc4b8a640 1 --2- 172.21.15.142:0/1433029088 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1dc01055b0 0x7f1dc019a680 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f1db4000c00 tx=0x7f1db4030b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:20:10.142 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:09.250+0000 7f1dbdffb640 1 -- 172.21.15.142:0/1433029088 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1db4037040 con 0x7f1dc01055b0 2024-09-06T14:20:10.142 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:09.250+0000 7f1dbdffb640 1 -- 172.21.15.142:0/1433029088 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7f1db4040070 con 0x7f1dc01055b0 2024-09-06T14:20:10.142 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:09.250+0000 7f1dc6dee640 1 -- 172.21.15.142:0/1433029088 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1dc019ae70 con 0x7f1dc01055b0 2024-09-06T14:20:10.142 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:09.250+0000 7f1dbdffb640 1 -- 172.21.15.142:0/1433029088 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1db403a390 con 0x7f1dc01055b0 2024-09-06T14:20:10.142 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:09.250+0000 7f1dc6dee640 1 -- 172.21.15.142:0/1433029088 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1dc019b830 con 0x7f1dc01055b0 2024-09-06T14:20:10.142 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:09.251+0000 7f1dbdffb640 1 -- 172.21.15.142:0/1433029088 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 12) ==== 50088+0+0 (secure 0 0 0) 0x7f1db403a5c0 con 0x7f1dc01055b0 2024-09-06T14:20:10.142 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:09.251+0000 7f1dc6dee640 1 -- 172.21.15.142:0/1433029088 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1dc01059b0 con 0x7f1dc01055b0 2024-09-06T14:20:10.142 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:09.251+0000 7f1dbdffb640 1 --2- 172.21.15.142:0/1433029088 >> v2:172.21.15.142:6800/853729507 conn(0x7f1db003daa0 0x7f1db003ff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:20:10.142 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:09.251+0000 7f1dbdffb640 1 -- 172.21.15.142:0/1433029088 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 983+0+0 (secure 0 0 0) 0x7f1db4072100 con 0x7f1dc01055b0 2024-09-06T14:20:10.142 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:09.251+0000 7f1dbffff640 1 --2- 172.21.15.142:0/1433029088 >> v2:172.21.15.142:6800/853729507 conn(0x7f1db003daa0 0x7f1db003ff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:20:10.142 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:09.252+0000 7f1dbffff640 1 --2- 172.21.15.142:0/1433029088 >> v2:172.21.15.142:6800/853729507 conn(0x7f1db003daa0 0x7f1db003ff60 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f1dac0098c0 tx=0x7f1dac006db0 comp rx=0 tx=0).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:20:10.142 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:09.255+0000 7f1dbdffb640 1 -- 172.21.15.142:0/1433029088 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7f1db4039d90 con 0x7f1dc01055b0 2024-09-06T14:20:10.143 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:09.492+0000 7f1dc6dee640 1 -- 172.21.15.142:0/1433029088 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7f1dc019bf70 con 0x7f1dc01055b0 2024-09-06T14:20:10.143 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:09.495+0000 7f1dbdffb640 1 -- 172.21.15.142:0/1433029088 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v27)=0 set mgr/dashboard/cluster/status v27) ==== 153+0+0 (secure 0 0 0) 0x7f1db403a8b0 con 0x7f1dc01055b0 2024-09-06T14:20:10.143 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-09-06T14:20:10.143 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:09.497+0000 7f1dc6dee640 1 -- 172.21.15.142:0/1433029088 >> v2:172.21.15.142:6800/853729507 conn(0x7f1db003daa0 msgr2=0x7f1db003ff60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:20:10.143 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:09.497+0000 7f1dc6dee640 1 --2- 172.21.15.142:0/1433029088 >> v2:172.21.15.142:6800/853729507 conn(0x7f1db003daa0 0x7f1db003ff60 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f1dac0098c0 tx=0x7f1dac006db0 comp rx=0 tx=0).stop 2024-09-06T14:20:10.143 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:09.497+0000 7f1dc6dee640 1 -- 172.21.15.142:0/1433029088 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1dc01055b0 msgr2=0x7f1dc019a680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:20:10.143 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:09.497+0000 7f1dc6dee640 1 --2- 172.21.15.142:0/1433029088 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1dc01055b0 0x7f1dc019a680 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f1db4000c00 tx=0x7f1db4030b70 comp rx=0 tx=0).stop 2024-09-06T14:20:10.143 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:09.497+0000 7f1dc6dee640 1 -- 172.21.15.142:0/1433029088 shutdown_connections 2024-09-06T14:20:10.143 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:09.497+0000 7f1dc6dee640 1 --2- 172.21.15.142:0/1433029088 >> v2:172.21.15.142:6800/853729507 conn(0x7f1db003daa0 0x7f1db003ff60 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:20:10.143 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:09.497+0000 7f1dc6dee640 1 --2- 172.21.15.142:0/1433029088 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1dc01055b0 0x7f1dc019a680 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:20:10.143 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:09.497+0000 7f1dc6dee640 1 -- 172.21.15.142:0/1433029088 >> 172.21.15.142:0/1433029088 conn(0x7f1dc0100f40 msgr2=0x7f1dc0079020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:20:10.143 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:09.497+0000 7f1dc6dee640 1 -- 172.21.15.142:0/1433029088 shutdown_connections 2024-09-06T14:20:10.143 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/ceph: stderr 2024-09-06T14:20:09.497+0000 7f1dc6dee640 1 -- 172.21.15.142:0/1433029088 wait complete. 2024-09-06T14:20:10.144 INFO:teuthology.orchestra.run.smithi142.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-09-06T14:20:10.144 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:20:10.144 INFO:teuthology.orchestra.run.smithi142.stdout: sudo /sbin/cephadm shell --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-09-06T14:20:10.144 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:20:10.144 INFO:teuthology.orchestra.run.smithi142.stdout:Or, if you are only running a single cluster on this host: 2024-09-06T14:20:10.144 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:20:10.144 INFO:teuthology.orchestra.run.smithi142.stdout: sudo /sbin/cephadm shell 2024-09-06T14:20:10.144 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:20:10.144 INFO:teuthology.orchestra.run.smithi142.stdout:Please consider enabling telemetry to help improve Ceph: 2024-09-06T14:20:10.144 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:20:10.144 INFO:teuthology.orchestra.run.smithi142.stdout: ceph telemetry on 2024-09-06T14:20:10.144 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:20:10.144 INFO:teuthology.orchestra.run.smithi142.stdout:For more information see: 2024-09-06T14:20:10.144 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:20:10.145 INFO:teuthology.orchestra.run.smithi142.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-09-06T14:20:10.145 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:20:10.145 INFO:teuthology.orchestra.run.smithi142.stdout:Bootstrap complete. 2024-09-06T14:20:10.175 INFO:tasks.cephadm:Fetching config... 2024-09-06T14:20:10.175 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-09-06T14:20:10.175 DEBUG:teuthology.orchestra.run.smithi142:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-09-06T14:20:10.191 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-09-06T14:20:10.192 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-09-06T14:20:10.192 DEBUG:teuthology.orchestra.run.smithi142:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-09-06T14:20:10.248 INFO:tasks.cephadm:Fetching mon keyring... 2024-09-06T14:20:10.249 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-09-06T14:20:10.249 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd if=/var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/keyring of=/dev/stdout 2024-09-06T14:20:10.313 INFO:tasks.cephadm:Fetching pub ssh key... 2024-09-06T14:20:10.313 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-09-06T14:20:10.314 DEBUG:teuthology.orchestra.run.smithi142:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-09-06T14:20:10.370 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-09-06T14:20:10.370 DEBUG:teuthology.orchestra.run.smithi142:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCUSzFmAPVYQutDQjVksjZD+8qjmKgJU9tbqlPuy65HiuDiUrrerefQP40TZVyEcrViBYVr81EQArBuusVmqCqGricoPa8W6qCsMmBkIXYCfsX0rzRjCgVJ/DfkPW8ljHm1TzJN/HHSHZTlcTcNu8FdmvzFa4qenA2NhuOwv1RSMGRqDAa3/5MKvMUeFcyZROG2rWW1WNVaAn0+yEUYxruisbuGktxa2okA8Qs4qARnMF5vTIMjADz0/Bp/Ds74KDr/L+eB90Rj2MJu3EADEAQbcgp2OGM39//mutyW2d+wI/XRGIl29tKWv3GiNpq2gyyfHtTZEwVcY4QlNHADia9YwLm0XsH1pwpf9n92I+nwLq6A/Id0reAvLDbLjdCRx6VUapqfW2r0IZ0969jTPpDAoCiR9FyH633ambNPua6pAiLiiuaDlSjPwwAObaU2eamplgyyEEPJKPoCcSiZ/QfM73lzPzMb8rzUV+NL7MMF0rjBgvMYfnvERjhzkla4fFU= ceph-ce32462e-6c5a-11ef-bce4-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-06T14:20:10.444 INFO:teuthology.orchestra.run.smithi142.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCUSzFmAPVYQutDQjVksjZD+8qjmKgJU9tbqlPuy65HiuDiUrrerefQP40TZVyEcrViBYVr81EQArBuusVmqCqGricoPa8W6qCsMmBkIXYCfsX0rzRjCgVJ/DfkPW8ljHm1TzJN/HHSHZTlcTcNu8FdmvzFa4qenA2NhuOwv1RSMGRqDAa3/5MKvMUeFcyZROG2rWW1WNVaAn0+yEUYxruisbuGktxa2okA8Qs4qARnMF5vTIMjADz0/Bp/Ds74KDr/L+eB90Rj2MJu3EADEAQbcgp2OGM39//mutyW2d+wI/XRGIl29tKWv3GiNpq2gyyfHtTZEwVcY4QlNHADia9YwLm0XsH1pwpf9n92I+nwLq6A/Id0reAvLDbLjdCRx6VUapqfW2r0IZ0969jTPpDAoCiR9FyH633ambNPua6pAiLiiuaDlSjPwwAObaU2eamplgyyEEPJKPoCcSiZ/QfM73lzPzMb8rzUV+NL7MMF0rjBgvMYfnvERjhzkla4fFU= ceph-ce32462e-6c5a-11ef-bce4-c7b262605968 2024-09-06T14:20:10.455 DEBUG:teuthology.orchestra.run.smithi162:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCUSzFmAPVYQutDQjVksjZD+8qjmKgJU9tbqlPuy65HiuDiUrrerefQP40TZVyEcrViBYVr81EQArBuusVmqCqGricoPa8W6qCsMmBkIXYCfsX0rzRjCgVJ/DfkPW8ljHm1TzJN/HHSHZTlcTcNu8FdmvzFa4qenA2NhuOwv1RSMGRqDAa3/5MKvMUeFcyZROG2rWW1WNVaAn0+yEUYxruisbuGktxa2okA8Qs4qARnMF5vTIMjADz0/Bp/Ds74KDr/L+eB90Rj2MJu3EADEAQbcgp2OGM39//mutyW2d+wI/XRGIl29tKWv3GiNpq2gyyfHtTZEwVcY4QlNHADia9YwLm0XsH1pwpf9n92I+nwLq6A/Id0reAvLDbLjdCRx6VUapqfW2r0IZ0969jTPpDAoCiR9FyH633ambNPua6pAiLiiuaDlSjPwwAObaU2eamplgyyEEPJKPoCcSiZ/QfM73lzPzMb8rzUV+NL7MMF0rjBgvMYfnvERjhzkla4fFU= ceph-ce32462e-6c5a-11ef-bce4-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-06T14:20:10.508 INFO:teuthology.orchestra.run.smithi162.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCUSzFmAPVYQutDQjVksjZD+8qjmKgJU9tbqlPuy65HiuDiUrrerefQP40TZVyEcrViBYVr81EQArBuusVmqCqGricoPa8W6qCsMmBkIXYCfsX0rzRjCgVJ/DfkPW8ljHm1TzJN/HHSHZTlcTcNu8FdmvzFa4qenA2NhuOwv1RSMGRqDAa3/5MKvMUeFcyZROG2rWW1WNVaAn0+yEUYxruisbuGktxa2okA8Qs4qARnMF5vTIMjADz0/Bp/Ds74KDr/L+eB90Rj2MJu3EADEAQbcgp2OGM39//mutyW2d+wI/XRGIl29tKWv3GiNpq2gyyfHtTZEwVcY4QlNHADia9YwLm0XsH1pwpf9n92I+nwLq6A/Id0reAvLDbLjdCRx6VUapqfW2r0IZ0969jTPpDAoCiR9FyH633ambNPua6pAiLiiuaDlSjPwwAObaU2eamplgyyEEPJKPoCcSiZ/QfM73lzPzMb8rzUV+NL7MMF0rjBgvMYfnvERjhzkla4fFU= ceph-ce32462e-6c5a-11ef-bce4-c7b262605968 2024-09-06T14:20:10.524 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-09-06T14:20:10.707 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:10 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/1433029088' entity='client.admin' 2024-09-06T14:20:10.792 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:20:11.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:11.302+0000 7fd5f64d3640 1 -- 172.21.15.142:0/1213203651 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd5f0101050 msgr2=0x7fd5f0101450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:20:11.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:11.302+0000 7fd5f64d3640 1 --2- 172.21.15.142:0/1213203651 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd5f0101050 0x7fd5f0101450 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fd5d8009880 tx=0x7fd5d802a5d0 comp rx=0 tx=0).stop 2024-09-06T14:20:11.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:11.302+0000 7fd5f64d3640 1 -- 172.21.15.142:0/1213203651 shutdown_connections 2024-09-06T14:20:11.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:11.302+0000 7fd5f64d3640 1 --2- 172.21.15.142:0/1213203651 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd5f0101050 0x7fd5f0101450 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:20:11.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:11.302+0000 7fd5f64d3640 1 -- 172.21.15.142:0/1213203651 >> 172.21.15.142:0/1213203651 conn(0x7fd5f00fc920 msgr2=0x7fd5f00fed60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:20:11.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:11.302+0000 7fd5f64d3640 1 -- 172.21.15.142:0/1213203651 shutdown_connections 2024-09-06T14:20:11.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:11.302+0000 7fd5f64d3640 1 -- 172.21.15.142:0/1213203651 wait complete. 2024-09-06T14:20:11.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:11.302+0000 7fd5f64d3640 1 Processor -- start 2024-09-06T14:20:11.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:11.302+0000 7fd5f64d3640 1 -- start start 2024-09-06T14:20:11.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:11.303+0000 7fd5f64d3640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd5f0101050 0x7fd5f006abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:20:11.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:11.303+0000 7fd5f64d3640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fd5f0106760 con 0x7fd5f0101050 2024-09-06T14:20:11.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:11.303+0000 7fd5effff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd5f0101050 0x7fd5f006abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:20:11.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:11.303+0000 7fd5effff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd5f0101050 0x7fd5f006abe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:41194/0 (socket says 172.21.15.142:41194) 2024-09-06T14:20:11.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:11.303+0000 7fd5effff640 1 -- 172.21.15.142:0/2342550332 learned_addr learned my addr 172.21.15.142:0/2342550332 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:20:11.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:11.303+0000 7fd5effff640 1 -- 172.21.15.142:0/2342550332 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5f006b140 con 0x7fd5f0101050 2024-09-06T14:20:11.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:11.303+0000 7fd5effff640 1 --2- 172.21.15.142:0/2342550332 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd5f0101050 0x7fd5f006abe0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd5d80057c0 tx=0x7fd5d80048f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:20:11.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:11.303+0000 7fd5ed7fa640 1 -- 172.21.15.142:0/2342550332 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd5d803f070 con 0x7fd5f0101050 2024-09-06T14:20:11.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:11.303+0000 7fd5ed7fa640 1 -- 172.21.15.142:0/2342550332 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7fd5d8003ba0 con 0x7fd5f0101050 2024-09-06T14:20:11.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:11.303+0000 7fd5f64d3640 1 -- 172.21.15.142:0/2342550332 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5f0199de0 con 0x7fd5f0101050 2024-09-06T14:20:11.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:11.303+0000 7fd5ed7fa640 1 -- 172.21.15.142:0/2342550332 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd5d8035030 con 0x7fd5f0101050 2024-09-06T14:20:11.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:11.303+0000 7fd5f64d3640 1 -- 172.21.15.142:0/2342550332 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5f019a2b0 con 0x7fd5f0101050 2024-09-06T14:20:11.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:11.304+0000 7fd5ed7fa640 1 -- 172.21.15.142:0/2342550332 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 12) ==== 50088+0+0 (secure 0 0 0) 0x7fd5d8042070 con 0x7fd5f0101050 2024-09-06T14:20:11.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:11.304+0000 7fd5f64d3640 1 -- 172.21.15.142:0/2342550332 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5f0106ee0 con 0x7fd5f0101050 2024-09-06T14:20:11.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:11.305+0000 7fd5ed7fa640 1 --2- 172.21.15.142:0/2342550332 >> v2:172.21.15.142:6800/853729507 conn(0x7fd5dc03daa0 0x7fd5dc03ff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:20:11.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:11.305+0000 7fd5ed7fa640 1 -- 172.21.15.142:0/2342550332 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 983+0+0 (secure 0 0 0) 0x7fd5d80727a0 con 0x7fd5f0101050 2024-09-06T14:20:11.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:11.305+0000 7fd5ef7fe640 1 --2- 172.21.15.142:0/2342550332 >> v2:172.21.15.142:6800/853729507 conn(0x7fd5dc03daa0 0x7fd5dc03ff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:20:11.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:11.305+0000 7fd5ef7fe640 1 --2- 172.21.15.142:0/2342550332 >> v2:172.21.15.142:6800/853729507 conn(0x7fd5dc03daa0 0x7fd5dc03ff60 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fd5e00098c0 tx=0x7fd5e0006db0 comp rx=0 tx=0).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:20:11.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:11.309+0000 7fd5ed7fa640 1 -- 172.21.15.142:0/2342550332 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7fd5d8040df0 con 0x7fd5f0101050 2024-09-06T14:20:11.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:11.481+0000 7fd5f64d3640 1 -- 172.21.15.142:0/2342550332 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7fd5f019a8d0 con 0x7fd5f0101050 2024-09-06T14:20:11.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:11.484+0000 7fd5ed7fa640 1 -- 172.21.15.142:0/2342550332 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7fd5d803b460 con 0x7fd5f0101050 2024-09-06T14:20:11.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:11.486+0000 7fd5f64d3640 1 -- 172.21.15.142:0/2342550332 >> v2:172.21.15.142:6800/853729507 conn(0x7fd5dc03daa0 msgr2=0x7fd5dc03ff60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:20:11.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:11.486+0000 7fd5f64d3640 1 --2- 172.21.15.142:0/2342550332 >> v2:172.21.15.142:6800/853729507 conn(0x7fd5dc03daa0 0x7fd5dc03ff60 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fd5e00098c0 tx=0x7fd5e0006db0 comp rx=0 tx=0).stop 2024-09-06T14:20:11.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:11.486+0000 7fd5f64d3640 1 -- 172.21.15.142:0/2342550332 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd5f0101050 msgr2=0x7fd5f006abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:20:11.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:11.486+0000 7fd5f64d3640 1 --2- 172.21.15.142:0/2342550332 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd5f0101050 0x7fd5f006abe0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd5d80057c0 tx=0x7fd5d80048f0 comp rx=0 tx=0).stop 2024-09-06T14:20:11.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:11.486+0000 7fd5f64d3640 1 -- 172.21.15.142:0/2342550332 shutdown_connections 2024-09-06T14:20:11.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:11.486+0000 7fd5f64d3640 1 --2- 172.21.15.142:0/2342550332 >> v2:172.21.15.142:6800/853729507 conn(0x7fd5dc03daa0 0x7fd5dc03ff60 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:20:11.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:11.486+0000 7fd5f64d3640 1 --2- 172.21.15.142:0/2342550332 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd5f0101050 0x7fd5f006abe0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:20:11.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:11.486+0000 7fd5f64d3640 1 -- 172.21.15.142:0/2342550332 >> 172.21.15.142:0/2342550332 conn(0x7fd5f00fc920 msgr2=0x7fd5f00fd570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:20:11.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:11.486+0000 7fd5f64d3640 1 -- 172.21.15.142:0/2342550332 shutdown_connections 2024-09-06T14:20:11.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:11.486+0000 7fd5f64d3640 1 -- 172.21.15.142:0/2342550332 wait complete. 2024-09-06T14:20:12.512 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:12 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/2342550332' entity='client.admin' 2024-09-06T14:20:12.512 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:12 smithi142 ceph-mon[29958]: mgrmap e13: y(active, since 12s) 2024-09-06T14:20:12.512 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:12 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:12.513 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:12 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:12.513 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:12 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd/host:smithi142", "name": "osd_memory_target"} : dispatch 2024-09-06T14:20:12.513 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:12 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:12.513 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:12 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-06T14:20:12.513 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:12 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:12.539 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-09-06T14:20:12.540 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-09-06T14:20:12.780 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:20:13.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:13.247+0000 7f6fb0d53640 1 -- 172.21.15.142:0/964675792 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6fac102180 msgr2=0x7f6fac102580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:20:13.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:13.247+0000 7f6fb0d53640 1 --2- 172.21.15.142:0/964675792 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6fac102180 0x7f6fac102580 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f6f9c009880 tx=0x7f6f9c02a5d0 comp rx=0 tx=0).stop 2024-09-06T14:20:13.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:13.247+0000 7f6fb0d53640 1 -- 172.21.15.142:0/964675792 shutdown_connections 2024-09-06T14:20:13.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:13.247+0000 7f6fb0d53640 1 --2- 172.21.15.142:0/964675792 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6fac102180 0x7f6fac102580 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:20:13.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:13.247+0000 7f6fb0d53640 1 -- 172.21.15.142:0/964675792 >> 172.21.15.142:0/964675792 conn(0x7f6fac0fda70 msgr2=0x7f6fac0ffe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:20:13.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:13.247+0000 7f6fb0d53640 1 -- 172.21.15.142:0/964675792 shutdown_connections 2024-09-06T14:20:13.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:13.247+0000 7f6fb0d53640 1 -- 172.21.15.142:0/964675792 wait complete. 2024-09-06T14:20:13.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:13.248+0000 7f6fb0d53640 1 Processor -- start 2024-09-06T14:20:13.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:13.248+0000 7f6fb0d53640 1 -- start start 2024-09-06T14:20:13.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:13.248+0000 7f6fb0d53640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6fac102180 0x7f6fac1970b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:20:13.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:13.248+0000 7f6fb0d53640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f6fac107890 con 0x7f6fac102180 2024-09-06T14:20:13.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:13.249+0000 7f6faa59c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6fac102180 0x7f6fac1970b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:20:13.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:13.249+0000 7f6faa59c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6fac102180 0x7f6fac1970b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:41212/0 (socket says 172.21.15.142:41212) 2024-09-06T14:20:13.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:13.249+0000 7f6faa59c640 1 -- 172.21.15.142:0/250607516 learned_addr learned my addr 172.21.15.142:0/250607516 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:20:13.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:13.249+0000 7f6faa59c640 1 -- 172.21.15.142:0/250607516 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fac197610 con 0x7f6fac102180 2024-09-06T14:20:13.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:13.249+0000 7f6faa59c640 1 --2- 172.21.15.142:0/250607516 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6fac102180 0x7f6fac1970b0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f6f9c0057c0 tx=0x7f6f9c0048f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:20:13.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:13.249+0000 7f6fa3fff640 1 -- 172.21.15.142:0/250607516 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6f9c03f070 con 0x7f6fac102180 2024-09-06T14:20:13.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:13.249+0000 7f6fa3fff640 1 -- 172.21.15.142:0/250607516 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7f6f9c003ba0 con 0x7f6fac102180 2024-09-06T14:20:13.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:13.249+0000 7f6fb0d53640 1 -- 172.21.15.142:0/250607516 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6fac1978a0 con 0x7f6fac102180 2024-09-06T14:20:13.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:13.249+0000 7f6fa3fff640 1 -- 172.21.15.142:0/250607516 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6f9c035030 con 0x7f6fac102180 2024-09-06T14:20:13.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:13.249+0000 7f6fb0d53640 1 -- 172.21.15.142:0/250607516 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6fac198260 con 0x7f6fac102180 2024-09-06T14:20:13.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:13.250+0000 7f6fa3fff640 1 -- 172.21.15.142:0/250607516 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 13) ==== 50132+0+0 (secure 0 0 0) 0x7f6f9c042070 con 0x7f6fac102180 2024-09-06T14:20:13.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:13.250+0000 7f6fa3fff640 1 --2- 172.21.15.142:0/250607516 >> v2:172.21.15.142:6800/853729507 conn(0x7f6f9803db40 0x7f6f98040000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:20:13.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:13.250+0000 7f6fb0d53640 1 -- 172.21.15.142:0/250607516 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6fac108010 con 0x7f6fac102180 2024-09-06T14:20:13.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:13.250+0000 7f6fa3fff640 1 -- 172.21.15.142:0/250607516 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 983+0+0 (secure 0 0 0) 0x7f6f9c0728d0 con 0x7f6fac102180 2024-09-06T14:20:13.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:13.250+0000 7f6fa1d9b640 1 --2- 172.21.15.142:0/250607516 >> v2:172.21.15.142:6800/853729507 conn(0x7f6f9803db40 0x7f6f98040000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:20:13.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:13.251+0000 7f6fa1d9b640 1 --2- 172.21.15.142:0/250607516 >> v2:172.21.15.142:6800/853729507 conn(0x7f6f9803db40 0x7f6f98040000 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f6f940098c0 tx=0x7f6f94006db0 comp rx=0 tx=0).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:20:13.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:13.255+0000 7f6fa3fff640 1 -- 172.21.15.142:0/250607516 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7f6f9c039df0 con 0x7f6fac102180 2024-09-06T14:20:13.403 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:13.408+0000 7f6fb0d53640 1 -- 172.21.15.142:0/250607516 --> v2:172.21.15.142:6800/853729507 -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7f6fac0ffd90 con 0x7f6f9803db40 2024-09-06T14:20:13.406 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:13.411+0000 7f6fa3fff640 1 -- 172.21.15.142:0/250607516 <== mgr.14146 v2:172.21.15.142:6800/853729507 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f6fac0ffd90 con 0x7f6f9803db40 2024-09-06T14:20:13.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:13.413+0000 7f6fb0d53640 1 -- 172.21.15.142:0/250607516 >> v2:172.21.15.142:6800/853729507 conn(0x7f6f9803db40 msgr2=0x7f6f98040000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:20:13.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:13.413+0000 7f6fb0d53640 1 --2- 172.21.15.142:0/250607516 >> v2:172.21.15.142:6800/853729507 conn(0x7f6f9803db40 0x7f6f98040000 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f6f940098c0 tx=0x7f6f94006db0 comp rx=0 tx=0).stop 2024-09-06T14:20:13.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:13.413+0000 7f6fb0d53640 1 -- 172.21.15.142:0/250607516 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6fac102180 msgr2=0x7f6fac1970b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:20:13.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:13.413+0000 7f6fb0d53640 1 --2- 172.21.15.142:0/250607516 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6fac102180 0x7f6fac1970b0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f6f9c0057c0 tx=0x7f6f9c0048f0 comp rx=0 tx=0).stop 2024-09-06T14:20:13.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:13.414+0000 7f6fb0d53640 1 -- 172.21.15.142:0/250607516 shutdown_connections 2024-09-06T14:20:13.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:13.414+0000 7f6fb0d53640 1 --2- 172.21.15.142:0/250607516 >> v2:172.21.15.142:6800/853729507 conn(0x7f6f9803db40 0x7f6f98040000 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:20:13.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:13.414+0000 7f6fb0d53640 1 --2- 172.21.15.142:0/250607516 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6fac102180 0x7f6fac1970b0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:20:13.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:13.414+0000 7f6fb0d53640 1 -- 172.21.15.142:0/250607516 >> 172.21.15.142:0/250607516 conn(0x7f6fac0fda70 msgr2=0x7f6fac0fe670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:20:13.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:13.414+0000 7f6fb0d53640 1 -- 172.21.15.142:0/250607516 shutdown_connections 2024-09-06T14:20:13.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:13.414+0000 7f6fb0d53640 1 -- 172.21.15.142:0/250607516 wait complete. 2024-09-06T14:20:14.185 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi162 2024-09-06T14:20:14.185 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-09-06T14:20:14.186 DEBUG:teuthology.orchestra.run.smithi162:> dd of=/etc/ceph/ceph.conf 2024-09-06T14:20:14.205 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-09-06T14:20:14.205 DEBUG:teuthology.orchestra.run.smithi162:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-06T14:20:14.264 INFO:tasks.cephadm:Adding host smithi162 to orchestrator... 2024-09-06T14:20:14.264 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph orch host add smithi162 2024-09-06T14:20:14.457 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:14 smithi142 ceph-mon[29958]: from='client.14170 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-09-06T14:20:14.457 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:14 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:14.457 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:14 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-06T14:20:14.457 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:14 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:20:14.457 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:14 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-06T14:20:14.457 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:14 smithi142 ceph-mon[29958]: Updating smithi142:/etc/ceph/ceph.conf 2024-09-06T14:20:14.457 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:14 smithi142 ceph-mon[29958]: Updating smithi142:/var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/config/ceph.conf 2024-09-06T14:20:14.457 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:14 smithi142 ceph-mon[29958]: Updating smithi142:/etc/ceph/ceph.client.admin.keyring 2024-09-06T14:20:14.457 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:14 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:14.457 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:14 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:14.458 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:14 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:14.506 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:20:14.974 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:14.969+0000 7fe87d225640 1 -- 172.21.15.142:0/3331926548 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe8781013e0 msgr2=0x7fe8781017c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:20:14.974 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:14.969+0000 7fe87d225640 1 --2- 172.21.15.142:0/3331926548 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe8781013e0 0x7fe8781017c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe868009880 tx=0x7fe86802a5d0 comp rx=0 tx=0).stop 2024-09-06T14:20:14.974 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:14.970+0000 7fe87d225640 1 -- 172.21.15.142:0/3331926548 shutdown_connections 2024-09-06T14:20:14.974 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:14.970+0000 7fe87d225640 1 --2- 172.21.15.142:0/3331926548 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe8781013e0 0x7fe8781017c0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:20:14.975 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:14.970+0000 7fe87d225640 1 -- 172.21.15.142:0/3331926548 >> 172.21.15.142:0/3331926548 conn(0x7fe8780fce90 msgr2=0x7fe8780ff2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:20:14.975 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:14.970+0000 7fe87d225640 1 -- 172.21.15.142:0/3331926548 shutdown_connections 2024-09-06T14:20:14.975 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:14.970+0000 7fe87d225640 1 -- 172.21.15.142:0/3331926548 wait complete. 2024-09-06T14:20:14.975 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:14.970+0000 7fe87d225640 1 Processor -- start 2024-09-06T14:20:14.975 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:14.971+0000 7fe87d225640 1 -- start start 2024-09-06T14:20:14.975 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:14.971+0000 7fe87d225640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe8781013e0 0x7fe878198260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:20:14.975 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:14.971+0000 7fe87d225640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fe87810ad40 con 0x7fe8781013e0 2024-09-06T14:20:14.975 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:14.971+0000 7fe876d9d640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe8781013e0 0x7fe878198260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:20:14.975 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:14.971+0000 7fe876d9d640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe8781013e0 0x7fe878198260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:41226/0 (socket says 172.21.15.142:41226) 2024-09-06T14:20:14.975 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:14.971+0000 7fe876d9d640 1 -- 172.21.15.142:0/3216168185 learned_addr learned my addr 172.21.15.142:0/3216168185 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:20:14.975 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:14.971+0000 7fe876d9d640 1 -- 172.21.15.142:0/3216168185 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe87806ba50 con 0x7fe8781013e0 2024-09-06T14:20:14.975 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:14.971+0000 7fe876d9d640 1 --2- 172.21.15.142:0/3216168185 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe8781013e0 0x7fe878198260 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fe868005ab0 tx=0x7fe8680048f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:20:14.975 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:14.971+0000 7fe85ffff640 1 -- 172.21.15.142:0/3216168185 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe86803f070 con 0x7fe8781013e0 2024-09-06T14:20:14.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:14.971+0000 7fe85ffff640 1 -- 172.21.15.142:0/3216168185 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7fe868003a70 con 0x7fe8781013e0 2024-09-06T14:20:14.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:14.971+0000 7fe87d225640 1 -- 172.21.15.142:0/3216168185 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8781987a0 con 0x7fe8781013e0 2024-09-06T14:20:14.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:14.971+0000 7fe85ffff640 1 -- 172.21.15.142:0/3216168185 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe868035030 con 0x7fe8781013e0 2024-09-06T14:20:14.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:14.971+0000 7fe87d225640 1 -- 172.21.15.142:0/3216168185 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe878198d00 con 0x7fe8781013e0 2024-09-06T14:20:14.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:14.972+0000 7fe85ffff640 1 -- 172.21.15.142:0/3216168185 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 13) ==== 50132+0+0 (secure 0 0 0) 0x7fe868042070 con 0x7fe8781013e0 2024-09-06T14:20:14.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:14.972+0000 7fe87d225640 1 -- 172.21.15.142:0/3216168185 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe878102690 con 0x7fe8781013e0 2024-09-06T14:20:14.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:14.973+0000 7fe85ffff640 1 --2- 172.21.15.142:0/3216168185 >> v2:172.21.15.142:6800/853729507 conn(0x7fe86003daf0 0x7fe86003ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:20:14.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:14.973+0000 7fe85ffff640 1 -- 172.21.15.142:0/3216168185 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 983+0+0 (secure 0 0 0) 0x7fe868072700 con 0x7fe8781013e0 2024-09-06T14:20:14.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:14.973+0000 7fe87659c640 1 --2- 172.21.15.142:0/3216168185 >> v2:172.21.15.142:6800/853729507 conn(0x7fe86003daf0 0x7fe86003ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:20:14.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:14.973+0000 7fe87659c640 1 --2- 172.21.15.142:0/3216168185 >> v2:172.21.15.142:6800/853729507 conn(0x7fe86003daf0 0x7fe86003ffb0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fe86c0098c0 tx=0x7fe86c006db0 comp rx=0 tx=0).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:20:14.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:14.977+0000 7fe85ffff640 1 -- 172.21.15.142:0/3216168185 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7fe868041e20 con 0x7fe8781013e0 2024-09-06T14:20:15.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:15.132+0000 7fe87d225640 1 -- 172.21.15.142:0/3216168185 --> v2:172.21.15.142:6800/853729507 -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi162", "target": ["mon-mgr", ""]}) -- 0x7fe87806afb0 con 0x7fe86003daf0 2024-09-06T14:20:15.706 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:15 smithi142 ceph-mon[29958]: Updating smithi142:/var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/config/ceph.client.admin.keyring 2024-09-06T14:20:16.707 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:16 smithi142 ceph-mon[29958]: from='client.14172 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi162", "target": ["mon-mgr", ""]}]: dispatch 2024-09-06T14:20:17.707 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:17 smithi142 ceph-mon[29958]: Deploying cephadm binary to smithi162 2024-09-06T14:20:18.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:18.260+0000 7fe85ffff640 1 -- 172.21.15.142:0/3216168185 <== mgr.14146 v2:172.21.15.142:6800/853729507 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7fe87806afb0 con 0x7fe86003daf0 2024-09-06T14:20:18.255 INFO:teuthology.orchestra.run.smithi142.stdout:Added host 'smithi162' with addr '172.21.15.162' 2024-09-06T14:20:18.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:18.262+0000 7fe87d225640 1 -- 172.21.15.142:0/3216168185 >> v2:172.21.15.142:6800/853729507 conn(0x7fe86003daf0 msgr2=0x7fe86003ffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:20:18.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:18.262+0000 7fe87d225640 1 --2- 172.21.15.142:0/3216168185 >> v2:172.21.15.142:6800/853729507 conn(0x7fe86003daf0 0x7fe86003ffb0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fe86c0098c0 tx=0x7fe86c006db0 comp rx=0 tx=0).stop 2024-09-06T14:20:18.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:18.262+0000 7fe87d225640 1 -- 172.21.15.142:0/3216168185 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe8781013e0 msgr2=0x7fe878198260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:20:18.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:18.262+0000 7fe87d225640 1 --2- 172.21.15.142:0/3216168185 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe8781013e0 0x7fe878198260 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fe868005ab0 tx=0x7fe8680048f0 comp rx=0 tx=0).stop 2024-09-06T14:20:18.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:18.262+0000 7fe87d225640 1 -- 172.21.15.142:0/3216168185 shutdown_connections 2024-09-06T14:20:18.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:18.262+0000 7fe87d225640 1 --2- 172.21.15.142:0/3216168185 >> v2:172.21.15.142:6800/853729507 conn(0x7fe86003daf0 0x7fe86003ffb0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:20:18.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:18.262+0000 7fe87d225640 1 --2- 172.21.15.142:0/3216168185 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe8781013e0 0x7fe878198260 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:20:18.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:18.262+0000 7fe87d225640 1 -- 172.21.15.142:0/3216168185 >> 172.21.15.142:0/3216168185 conn(0x7fe8780fce90 msgr2=0x7fe8781033a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:20:18.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:18.262+0000 7fe87d225640 1 -- 172.21.15.142:0/3216168185 shutdown_connections 2024-09-06T14:20:18.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:18.262+0000 7fe87d225640 1 -- 172.21.15.142:0/3216168185 wait complete. 2024-09-06T14:20:18.890 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph orch host ls --format=json 2024-09-06T14:20:19.130 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:20:19.505 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:19 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:19.505 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:19 smithi142 ceph-mon[29958]: Added host smithi162 2024-09-06T14:20:19.505 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:19 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-06T14:20:19.505 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:19 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:19.505 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:19 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:19.505 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:19 smithi142 ceph-mon[29958]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-06T14:20:19.506 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:19 smithi142 ceph-mon[29958]: Health check failed: Module 'volumes' has failed dependency: No module named 'ceph.fs' (MGR_MODULE_DEPENDENCY) 2024-09-06T14:20:19.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:19.590+0000 7f1646e7b640 1 -- 172.21.15.142:0/4042434788 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1640101420 msgr2=0x7f1640101800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:20:19.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:19.590+0000 7f1646e7b640 1 --2- 172.21.15.142:0/4042434788 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1640101420 0x7f1640101800 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f1634009880 tx=0x7f163402a5d0 comp rx=0 tx=0).stop 2024-09-06T14:20:19.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:19.590+0000 7f1646e7b640 1 -- 172.21.15.142:0/4042434788 shutdown_connections 2024-09-06T14:20:19.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:19.590+0000 7f1646e7b640 1 --2- 172.21.15.142:0/4042434788 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1640101420 0x7f1640101800 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:20:19.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:19.590+0000 7f1646e7b640 1 -- 172.21.15.142:0/4042434788 >> 172.21.15.142:0/4042434788 conn(0x7f16400fce90 msgr2=0x7f16400ff2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:20:19.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:19.590+0000 7f1646e7b640 1 -- 172.21.15.142:0/4042434788 shutdown_connections 2024-09-06T14:20:19.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:19.590+0000 7f1646e7b640 1 -- 172.21.15.142:0/4042434788 wait complete. 2024-09-06T14:20:19.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:19.590+0000 7f1646e7b640 1 Processor -- start 2024-09-06T14:20:19.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:19.590+0000 7f1646e7b640 1 -- start start 2024-09-06T14:20:19.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:19.590+0000 7f1646e7b640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1640101420 0x7f1640195240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:20:19.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:19.590+0000 7f1646e7b640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f164006b990 con 0x7f1640101420 2024-09-06T14:20:19.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:19.591+0000 7f1644c17640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1640101420 0x7f1640195240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:20:19.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:19.591+0000 7f1644c17640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1640101420 0x7f1640195240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:53156/0 (socket says 172.21.15.142:53156) 2024-09-06T14:20:19.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:19.591+0000 7f1644c17640 1 -- 172.21.15.142:0/1927006128 learned_addr learned my addr 172.21.15.142:0/1927006128 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:20:19.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:19.591+0000 7f1644c17640 1 -- 172.21.15.142:0/1927006128 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1640195780 con 0x7f1640101420 2024-09-06T14:20:19.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:19.591+0000 7f1644c17640 1 --2- 172.21.15.142:0/1927006128 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1640101420 0x7f1640195240 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1634005ab0 tx=0x7f16340048f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:20:19.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:19.591+0000 7f163dffb640 1 -- 172.21.15.142:0/1927006128 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f163403f070 con 0x7f1640101420 2024-09-06T14:20:19.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:19.591+0000 7f163dffb640 1 -- 172.21.15.142:0/1927006128 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7f1634003a70 con 0x7f1640101420 2024-09-06T14:20:19.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:19.591+0000 7f163dffb640 1 -- 172.21.15.142:0/1927006128 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1634035030 con 0x7f1640101420 2024-09-06T14:20:19.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:19.591+0000 7f1646e7b640 1 -- 172.21.15.142:0/1927006128 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1640195a70 con 0x7f1640101420 2024-09-06T14:20:19.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:19.591+0000 7f1646e7b640 1 -- 172.21.15.142:0/1927006128 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f164019ab20 con 0x7f1640101420 2024-09-06T14:20:19.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:19.592+0000 7f163dffb640 1 -- 172.21.15.142:0/1927006128 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 13) ==== 50132+0+0 (secure 0 0 0) 0x7f1634042070 con 0x7f1640101420 2024-09-06T14:20:19.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:19.592+0000 7f1646e7b640 1 -- 172.21.15.142:0/1927006128 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16401026d0 con 0x7f1640101420 2024-09-06T14:20:19.599 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:19.593+0000 7f163dffb640 1 --2- 172.21.15.142:0/1927006128 >> v2:172.21.15.142:6800/853729507 conn(0x7f163003fd90 0x7f1630042250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:20:19.599 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:19.593+0000 7f163dffb640 1 -- 172.21.15.142:0/1927006128 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 983+0+0 (secure 0 0 0) 0x7f16340725c0 con 0x7f1640101420 2024-09-06T14:20:19.599 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:19.593+0000 7f163ffff640 1 --2- 172.21.15.142:0/1927006128 >> v2:172.21.15.142:6800/853729507 conn(0x7f163003fd90 0x7f1630042250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:20:19.599 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:19.593+0000 7f163ffff640 1 --2- 172.21.15.142:0/1927006128 >> v2:172.21.15.142:6800/853729507 conn(0x7f163003fd90 0x7f1630042250 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f162c0098c0 tx=0x7f162c006db0 comp rx=0 tx=0).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:20:19.599 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:19.597+0000 7f163dffb640 1 -- 172.21.15.142:0/1927006128 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7f163404a260 con 0x7f1640101420 2024-09-06T14:20:19.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:19.756+0000 7f1646e7b640 1 -- 172.21.15.142:0/1927006128 --> v2:172.21.15.142:6800/853729507 -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f1640101880 con 0x7f163003fd90 2024-09-06T14:20:19.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:19.757+0000 7f163dffb640 1 -- 172.21.15.142:0/1927006128 <== mgr.14146 v2:172.21.15.142:6800/853729507 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+161 (secure 0 0 0) 0x7f1640101880 con 0x7f163003fd90 2024-09-06T14:20:19.752 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:20:19.752 INFO:teuthology.orchestra.run.smithi142.stdout:[{"addr": "172.21.15.142", "hostname": "smithi142", "labels": [], "status": ""}, {"addr": "172.21.15.162", "hostname": "smithi162", "labels": [], "status": ""}] 2024-09-06T14:20:19.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:19.758+0000 7f1646e7b640 1 -- 172.21.15.142:0/1927006128 >> v2:172.21.15.142:6800/853729507 conn(0x7f163003fd90 msgr2=0x7f1630042250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:20:19.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:19.758+0000 7f1646e7b640 1 --2- 172.21.15.142:0/1927006128 >> v2:172.21.15.142:6800/853729507 conn(0x7f163003fd90 0x7f1630042250 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f162c0098c0 tx=0x7f162c006db0 comp rx=0 tx=0).stop 2024-09-06T14:20:19.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:19.758+0000 7f1646e7b640 1 -- 172.21.15.142:0/1927006128 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1640101420 msgr2=0x7f1640195240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:20:19.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:19.758+0000 7f1646e7b640 1 --2- 172.21.15.142:0/1927006128 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1640101420 0x7f1640195240 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1634005ab0 tx=0x7f16340048f0 comp rx=0 tx=0).stop 2024-09-06T14:20:19.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:19.759+0000 7f1646e7b640 1 -- 172.21.15.142:0/1927006128 shutdown_connections 2024-09-06T14:20:19.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:19.759+0000 7f1646e7b640 1 --2- 172.21.15.142:0/1927006128 >> v2:172.21.15.142:6800/853729507 conn(0x7f163003fd90 0x7f1630042250 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:20:19.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:19.759+0000 7f1646e7b640 1 --2- 172.21.15.142:0/1927006128 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1640101420 0x7f1640195240 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:20:19.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:19.759+0000 7f1646e7b640 1 -- 172.21.15.142:0/1927006128 >> 172.21.15.142:0/1927006128 conn(0x7f16400fce90 msgr2=0x7f16401044d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:20:19.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:19.759+0000 7f1646e7b640 1 -- 172.21.15.142:0/1927006128 shutdown_connections 2024-09-06T14:20:19.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:19.759+0000 7f1646e7b640 1 -- 172.21.15.142:0/1927006128 wait complete. 2024-09-06T14:20:20.371 INFO:tasks.cephadm:Setting crush tunables to default 2024-09-06T14:20:20.372 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph osd crush tunables default 2024-09-06T14:20:20.612 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:20:20.957 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:20 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:20.963 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:20 smithi142 ceph-mon[29958]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-06T14:20:21.079 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:21.081+0000 7f4401fe6640 1 -- 172.21.15.142:0/1016773325 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f43fc1013e0 msgr2=0x7f43fc1017c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:20:21.079 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:21.081+0000 7f4401fe6640 1 --2- 172.21.15.142:0/1016773325 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f43fc1013e0 0x7f43fc1017c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f43ec009880 tx=0x7f43ec02a5d0 comp rx=0 tx=0).stop 2024-09-06T14:20:21.079 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:21.081+0000 7f4401fe6640 1 -- 172.21.15.142:0/1016773325 shutdown_connections 2024-09-06T14:20:21.080 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:21.081+0000 7f4401fe6640 1 --2- 172.21.15.142:0/1016773325 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f43fc1013e0 0x7f43fc1017c0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:20:21.080 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:21.081+0000 7f4401fe6640 1 -- 172.21.15.142:0/1016773325 >> 172.21.15.142:0/1016773325 conn(0x7f43fc0fce90 msgr2=0x7f43fc0ff2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:20:21.080 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:21.081+0000 7f4401fe6640 1 -- 172.21.15.142:0/1016773325 shutdown_connections 2024-09-06T14:20:21.080 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:21.081+0000 7f4401fe6640 1 -- 172.21.15.142:0/1016773325 wait complete. 2024-09-06T14:20:21.080 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:21.082+0000 7f4401fe6640 1 Processor -- start 2024-09-06T14:20:21.080 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:21.082+0000 7f4401fe6640 1 -- start start 2024-09-06T14:20:21.080 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:21.082+0000 7f4401fe6640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f43fc1013e0 0x7f43fc196340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:20:21.080 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:21.082+0000 7f4401fe6640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f43fc10ad40 con 0x7f43fc1013e0 2024-09-06T14:20:21.080 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:21.082+0000 7f43fb7fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f43fc1013e0 0x7f43fc196340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:20:21.080 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:21.082+0000 7f43fb7fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f43fc1013e0 0x7f43fc196340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:53174/0 (socket says 172.21.15.142:53174) 2024-09-06T14:20:21.080 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:21.082+0000 7f43fb7fe640 1 -- 172.21.15.142:0/552903435 learned_addr learned my addr 172.21.15.142:0/552903435 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:20:21.080 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:21.082+0000 7f43fb7fe640 1 -- 172.21.15.142:0/552903435 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43fc196880 con 0x7f43fc1013e0 2024-09-06T14:20:21.081 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:21.082+0000 7f43fb7fe640 1 --2- 172.21.15.142:0/552903435 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f43fc1013e0 0x7f43fc196340 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f43ec005ab0 tx=0x7f43ec0048f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:20:21.081 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:21.083+0000 7f43f97fa640 1 -- 172.21.15.142:0/552903435 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f43ec03f070 con 0x7f43fc1013e0 2024-09-06T14:20:21.081 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:21.083+0000 7f43f97fa640 1 -- 172.21.15.142:0/552903435 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7f43ec003a70 con 0x7f43fc1013e0 2024-09-06T14:20:21.081 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:21.083+0000 7f4401fe6640 1 -- 172.21.15.142:0/552903435 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43fc196b70 con 0x7f43fc1013e0 2024-09-06T14:20:21.081 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:21.083+0000 7f43f97fa640 1 -- 172.21.15.142:0/552903435 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f43ec035030 con 0x7f43fc1013e0 2024-09-06T14:20:21.081 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:21.083+0000 7f4401fe6640 1 -- 172.21.15.142:0/552903435 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43fc19bc20 con 0x7f43fc1013e0 2024-09-06T14:20:21.081 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:21.084+0000 7f43f97fa640 1 -- 172.21.15.142:0/552903435 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 13) ==== 50132+0+0 (secure 0 0 0) 0x7f43ec042070 con 0x7f43fc1013e0 2024-09-06T14:20:21.081 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:21.084+0000 7f4401fe6640 1 -- 172.21.15.142:0/552903435 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43fc102690 con 0x7f43fc1013e0 2024-09-06T14:20:21.081 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:21.084+0000 7f43f97fa640 1 --2- 172.21.15.142:0/552903435 >> v2:172.21.15.142:6800/853729507 conn(0x7f43e803db40 0x7f43e8040000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:20:21.081 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:21.084+0000 7f43f97fa640 1 -- 172.21.15.142:0/552903435 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 983+0+0 (secure 0 0 0) 0x7f43ec072650 con 0x7f43fc1013e0 2024-09-06T14:20:21.081 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:21.084+0000 7f43f3fff640 1 --2- 172.21.15.142:0/552903435 >> v2:172.21.15.142:6800/853729507 conn(0x7f43e803db40 0x7f43e8040000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:20:21.081 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:21.084+0000 7f43f3fff640 1 --2- 172.21.15.142:0/552903435 >> v2:172.21.15.142:6800/853729507 conn(0x7f43e803db40 0x7f43e8040000 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f43e40098c0 tx=0x7f43e4006db0 comp rx=0 tx=0).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:20:21.083 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:21.088+0000 7f43f97fa640 1 -- 172.21.15.142:0/552903435 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7f43ec04a260 con 0x7f43fc1013e0 2024-09-06T14:20:21.239 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:21.244+0000 7f4401fe6640 1 -- 172.21.15.142:0/552903435 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f43fc101840 con 0x7f43fc1013e0 2024-09-06T14:20:21.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:21.644+0000 7f43f97fa640 1 -- 172.21.15.142:0/552903435 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7f43ec0035f0 con 0x7f43fc1013e0 2024-09-06T14:20:21.640 INFO:teuthology.orchestra.run.smithi142.stderr:adjusted tunables profile to default 2024-09-06T14:20:21.642 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:21.647+0000 7f4401fe6640 1 -- 172.21.15.142:0/552903435 >> v2:172.21.15.142:6800/853729507 conn(0x7f43e803db40 msgr2=0x7f43e8040000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:20:21.642 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:21.647+0000 7f4401fe6640 1 --2- 172.21.15.142:0/552903435 >> v2:172.21.15.142:6800/853729507 conn(0x7f43e803db40 0x7f43e8040000 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f43e40098c0 tx=0x7f43e4006db0 comp rx=0 tx=0).stop 2024-09-06T14:20:21.642 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:21.647+0000 7f4401fe6640 1 -- 172.21.15.142:0/552903435 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f43fc1013e0 msgr2=0x7f43fc196340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:20:21.642 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:21.647+0000 7f4401fe6640 1 --2- 172.21.15.142:0/552903435 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f43fc1013e0 0x7f43fc196340 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f43ec005ab0 tx=0x7f43ec0048f0 comp rx=0 tx=0).stop 2024-09-06T14:20:21.642 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:21.647+0000 7f4401fe6640 1 -- 172.21.15.142:0/552903435 shutdown_connections 2024-09-06T14:20:21.642 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:21.647+0000 7f4401fe6640 1 --2- 172.21.15.142:0/552903435 >> v2:172.21.15.142:6800/853729507 conn(0x7f43e803db40 0x7f43e8040000 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:20:21.642 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:21.647+0000 7f4401fe6640 1 --2- 172.21.15.142:0/552903435 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f43fc1013e0 0x7f43fc196340 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:20:21.642 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:21.647+0000 7f4401fe6640 1 -- 172.21.15.142:0/552903435 >> 172.21.15.142:0/552903435 conn(0x7f43fc0fce90 msgr2=0x7f43fc1033a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:20:21.642 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:21.647+0000 7f4401fe6640 1 -- 172.21.15.142:0/552903435 shutdown_connections 2024-09-06T14:20:21.642 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:21.647+0000 7f4401fe6640 1 -- 172.21.15.142:0/552903435 wait complete. 2024-09-06T14:20:21.947 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:21 smithi142 ceph-mon[29958]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-06T14:20:21.948 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:21 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/552903435' entity='client.admin' cmd={"prefix": "osd crush tunables", "profile": "default"} : dispatch 2024-09-06T14:20:22.268 INFO:tasks.cephadm:Adding mon.a on smithi142 2024-09-06T14:20:22.269 INFO:tasks.cephadm:Adding mon.c on smithi142 2024-09-06T14:20:22.269 INFO:tasks.cephadm:Adding mon.b on smithi162 2024-09-06T14:20:22.269 DEBUG:teuthology.orchestra.run.smithi162:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph orch apply mon '3;smithi142:172.21.15.142=a;smithi142:[v2:172.21.15.142:3301,v1:172.21.15.142:6790]=c;smithi162:172.21.15.162=b' 2024-09-06T14:20:22.517 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-06T14:20:22.564 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-06T14:20:22.957 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:22 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/552903435' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-06T14:20:22.957 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:22 smithi142 ceph-mon[29958]: osdmap e4: 0 total, 0 up, 0 in 2024-09-06T14:20:22.957 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:22 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:22.957 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:22 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:22.957 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:22 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:22.957 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:22 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:22.957 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:22 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd/host:smithi162", "name": "osd_memory_target"} : dispatch 2024-09-06T14:20:22.957 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:22 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:20:22.957 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:22 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-06T14:20:23.085 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:23.058+0000 7fa0d8c18640 1 -- 172.21.15.162:0/1167578695 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa0d406af50 msgr2=0x7fa0d4104d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:20:23.085 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:23.058+0000 7fa0d8c18640 1 --2- 172.21.15.162:0/1167578695 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa0d406af50 0x7fa0d4104d90 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fa0c4009880 tx=0x7fa0c402a5d0 comp rx=0 tx=0).stop 2024-09-06T14:20:23.085 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:23.058+0000 7fa0d8c18640 1 -- 172.21.15.162:0/1167578695 shutdown_connections 2024-09-06T14:20:23.085 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:23.058+0000 7fa0d8c18640 1 --2- 172.21.15.162:0/1167578695 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa0d406af50 0x7fa0d4104d90 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:20:23.085 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:23.058+0000 7fa0d8c18640 1 -- 172.21.15.162:0/1167578695 >> 172.21.15.162:0/1167578695 conn(0x7fa0d40fc920 msgr2=0x7fa0d40fed60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:20:23.085 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:23.058+0000 7fa0d8c18640 1 -- 172.21.15.162:0/1167578695 shutdown_connections 2024-09-06T14:20:23.085 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:23.058+0000 7fa0d8c18640 1 -- 172.21.15.162:0/1167578695 wait complete. 2024-09-06T14:20:23.085 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:23.059+0000 7fa0d8c18640 1 Processor -- start 2024-09-06T14:20:23.085 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:23.059+0000 7fa0d8c18640 1 -- start start 2024-09-06T14:20:23.086 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:23.059+0000 7fa0d8c18640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa0d406af50 0x7fa0d4197090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:20:23.086 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:23.059+0000 7fa0d8c18640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fa0d4107850 con 0x7fa0d406af50 2024-09-06T14:20:23.086 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:23.060+0000 7fa0d259c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa0d406af50 0x7fa0d4197090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:20:23.086 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:23.060+0000 7fa0d259c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa0d406af50 0x7fa0d4197090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.162:44094/0 (socket says 172.21.15.162:44094) 2024-09-06T14:20:23.086 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:23.060+0000 7fa0d259c640 1 -- 172.21.15.162:0/3247684160 learned_addr learned my addr 172.21.15.162:0/3247684160 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-09-06T14:20:23.086 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:23.060+0000 7fa0d259c640 1 -- 172.21.15.162:0/3247684160 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0d41975f0 con 0x7fa0d406af50 2024-09-06T14:20:23.086 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:23.060+0000 7fa0d259c640 1 --2- 172.21.15.162:0/3247684160 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa0d406af50 0x7fa0d4197090 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa0c40057c0 tx=0x7fa0c40048f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:20:23.086 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:23.061+0000 7fa0c37fe640 1 -- 172.21.15.162:0/3247684160 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa0c403f070 con 0x7fa0d406af50 2024-09-06T14:20:23.086 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:23.061+0000 7fa0c37fe640 1 -- 172.21.15.162:0/3247684160 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7fa0c4003ba0 con 0x7fa0d406af50 2024-09-06T14:20:23.086 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:23.061+0000 7fa0c37fe640 1 -- 172.21.15.162:0/3247684160 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa0c4035030 con 0x7fa0d406af50 2024-09-06T14:20:23.086 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:23.061+0000 7fa0d8c18640 1 -- 172.21.15.162:0/3247684160 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0d4197880 con 0x7fa0d406af50 2024-09-06T14:20:23.086 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:23.061+0000 7fa0d8c18640 1 -- 172.21.15.162:0/3247684160 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0d4198240 con 0x7fa0d406af50 2024-09-06T14:20:23.086 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:23.062+0000 7fa0c37fe640 1 -- 172.21.15.162:0/3247684160 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 13) ==== 50132+0+0 (secure 0 0 0) 0x7fa0c4042070 con 0x7fa0d406af50 2024-09-06T14:20:23.087 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:23.062+0000 7fa0c37fe640 1 --2- 172.21.15.162:0/3247684160 >> v2:172.21.15.142:6800/853729507 conn(0x7fa0bc03db40 0x7fa0bc040000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:20:23.087 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:23.062+0000 7fa0c37fe640 1 -- 172.21.15.162:0/3247684160 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 983+0+0 (secure 0 0 0) 0x7fa0c4072850 con 0x7fa0d406af50 2024-09-06T14:20:23.087 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:23.062+0000 7fa0d8c18640 1 -- 172.21.15.162:0/3247684160 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0d4107fd0 con 0x7fa0d406af50 2024-09-06T14:20:23.087 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:23.062+0000 7fa0d1d9b640 1 --2- 172.21.15.162:0/3247684160 >> v2:172.21.15.142:6800/853729507 conn(0x7fa0bc03db40 0x7fa0bc040000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:20:23.087 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:23.062+0000 7fa0d1d9b640 1 --2- 172.21.15.162:0/3247684160 >> v2:172.21.15.142:6800/853729507 conn(0x7fa0bc03db40 0x7fa0bc040000 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fa0c80098c0 tx=0x7fa0c8006db0 comp rx=0 tx=0).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:20:23.087 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:23.067+0000 7fa0c37fe640 1 -- 172.21.15.162:0/3247684160 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7fa0c4039df0 con 0x7fa0d406af50 2024-09-06T14:20:23.237 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:23.241+0000 7fa0d8c18640 1 -- 172.21.15.162:0/3247684160 --> v2:172.21.15.142:6800/853729507 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi142:172.21.15.142=a;smithi142:[v2:172.21.15.142:3301,v1:172.21.15.142:6790]=c;smithi162:172.21.15.162=b", "target": ["mon-mgr", ""]}) -- 0x7fa0d40fed70 con 0x7fa0bc03db40 2024-09-06T14:20:23.246 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:23.246+0000 7fa0c37fe640 1 -- 172.21.15.162:0/3247684160 <== mgr.14146 v2:172.21.15.142:6800/853729507 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fa0d40fed70 con 0x7fa0bc03db40 2024-09-06T14:20:23.246 INFO:teuthology.orchestra.run.smithi162.stdout:Scheduled mon update... 2024-09-06T14:20:23.247 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:23.248+0000 7fa0d8c18640 1 -- 172.21.15.162:0/3247684160 >> v2:172.21.15.142:6800/853729507 conn(0x7fa0bc03db40 msgr2=0x7fa0bc040000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:20:23.247 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:23.248+0000 7fa0d8c18640 1 --2- 172.21.15.162:0/3247684160 >> v2:172.21.15.142:6800/853729507 conn(0x7fa0bc03db40 0x7fa0bc040000 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fa0c80098c0 tx=0x7fa0c8006db0 comp rx=0 tx=0).stop 2024-09-06T14:20:23.247 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:23.248+0000 7fa0d8c18640 1 -- 172.21.15.162:0/3247684160 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa0d406af50 msgr2=0x7fa0d4197090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:20:23.247 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:23.248+0000 7fa0d8c18640 1 --2- 172.21.15.162:0/3247684160 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa0d406af50 0x7fa0d4197090 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa0c40057c0 tx=0x7fa0c40048f0 comp rx=0 tx=0).stop 2024-09-06T14:20:23.247 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:23.248+0000 7fa0d8c18640 1 -- 172.21.15.162:0/3247684160 shutdown_connections 2024-09-06T14:20:23.247 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:23.248+0000 7fa0d8c18640 1 --2- 172.21.15.162:0/3247684160 >> v2:172.21.15.142:6800/853729507 conn(0x7fa0bc03db40 0x7fa0bc040000 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:20:23.247 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:23.248+0000 7fa0d8c18640 1 --2- 172.21.15.162:0/3247684160 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa0d406af50 0x7fa0d4197090 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:20:23.247 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:23.248+0000 7fa0d8c18640 1 -- 172.21.15.162:0/3247684160 >> 172.21.15.162:0/3247684160 conn(0x7fa0d40fc920 msgr2=0x7fa0d40fd570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:20:23.247 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:23.248+0000 7fa0d8c18640 1 -- 172.21.15.162:0/3247684160 shutdown_connections 2024-09-06T14:20:23.247 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:23.248+0000 7fa0d8c18640 1 -- 172.21.15.162:0/3247684160 wait complete. 2024-09-06T14:20:23.804 DEBUG:teuthology.orchestra.run.smithi142:mon.c> sudo journalctl -f -n 0 -u ceph-ce32462e-6c5a-11ef-bce4-c7b262605968@mon.c.service 2024-09-06T14:20:23.806 DEBUG:teuthology.orchestra.run.smithi162:mon.b> sudo journalctl -f -n 0 -u ceph-ce32462e-6c5a-11ef-bce4-c7b262605968@mon.b.service 2024-09-06T14:20:23.808 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-06T14:20:23.809 DEBUG:teuthology.orchestra.run.smithi162:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph mon dump -f json 2024-09-06T14:20:23.897 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:23 smithi142 ceph-mon[29958]: Updating smithi162:/etc/ceph/ceph.conf 2024-09-06T14:20:23.897 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:23 smithi142 ceph-mon[29958]: Updating smithi162:/var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/config/ceph.conf 2024-09-06T14:20:23.898 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:23 smithi142 ceph-mon[29958]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-06T14:20:23.898 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:23 smithi142 ceph-mon[29958]: Updating smithi162:/etc/ceph/ceph.client.admin.keyring 2024-09-06T14:20:23.898 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:23 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:23.898 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:23 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:23.898 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:23 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:23.898 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:23 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:23.898 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:23 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-06T14:20:23.898 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:23 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:20:24.530 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/config/ceph.conf 2024-09-06T14:20:24.957 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:24 smithi142 ceph-mon[29958]: Updating smithi162:/var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/config/ceph.client.admin.keyring 2024-09-06T14:20:24.957 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:24 smithi142 ceph-mon[29958]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi142:172.21.15.142=a;smithi142:[v2:172.21.15.142:3301,v1:172.21.15.142:6790]=c;smithi162:172.21.15.162=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-06T14:20:24.957 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:24 smithi142 ceph-mon[29958]: Saving service mon spec with placement smithi142:172.21.15.142=a;smithi142:[v2:172.21.15.142:3301,v1:172.21.15.142:6790]=c;smithi162:172.21.15.162=b;count:3 2024-09-06T14:20:24.957 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:24 smithi142 ceph-mon[29958]: Deploying daemon mon.b on smithi162 2024-09-06T14:20:25.290 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:25.282+0000 7fe4b20b7640 1 -- 172.21.15.162:0/574462577 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe4ac1011e0 msgr2=0x7fe4ac1015c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:20:25.290 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:25.282+0000 7fe4b20b7640 1 --2- 172.21.15.162:0/574462577 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe4ac1011e0 0x7fe4ac1015c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fe49c009880 tx=0x7fe49c02a5d0 comp rx=0 tx=0).stop 2024-09-06T14:20:25.290 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:25.282+0000 7fe4b20b7640 1 -- 172.21.15.162:0/574462577 shutdown_connections 2024-09-06T14:20:25.290 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:25.282+0000 7fe4b20b7640 1 --2- 172.21.15.162:0/574462577 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe4ac1011e0 0x7fe4ac1015c0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:20:25.290 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:25.282+0000 7fe4b20b7640 1 -- 172.21.15.162:0/574462577 >> 172.21.15.162:0/574462577 conn(0x7fe4ac0fce90 msgr2=0x7fe4ac0ff2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:20:25.290 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:25.282+0000 7fe4b20b7640 1 -- 172.21.15.162:0/574462577 shutdown_connections 2024-09-06T14:20:25.290 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:25.282+0000 7fe4b20b7640 1 -- 172.21.15.162:0/574462577 wait complete. 2024-09-06T14:20:25.290 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:25.283+0000 7fe4b20b7640 1 Processor -- start 2024-09-06T14:20:25.291 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:25.283+0000 7fe4b20b7640 1 -- start start 2024-09-06T14:20:25.291 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:25.283+0000 7fe4b20b7640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe4ac1011e0 0x7fe4ac19da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:20:25.291 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:25.283+0000 7fe4b20b7640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fe4ac112490 con 0x7fe4ac1011e0 2024-09-06T14:20:25.291 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:25.284+0000 7fe4ab7fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe4ac1011e0 0x7fe4ac19da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:20:25.291 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:25.284+0000 7fe4ab7fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe4ac1011e0 0x7fe4ac19da30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.162:44102/0 (socket says 172.21.15.162:44102) 2024-09-06T14:20:25.291 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:25.284+0000 7fe4ab7fe640 1 -- 172.21.15.162:0/1632527596 learned_addr learned my addr 172.21.15.162:0/1632527596 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-09-06T14:20:25.291 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:25.284+0000 7fe4ab7fe640 1 -- 172.21.15.162:0/1632527596 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4ac19df70 con 0x7fe4ac1011e0 2024-09-06T14:20:25.291 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:25.284+0000 7fe4ab7fe640 1 --2- 172.21.15.162:0/1632527596 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe4ac1011e0 0x7fe4ac19da30 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fe49c005ab0 tx=0x7fe49c0048f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:20:25.291 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:25.284+0000 7fe4a8ff9640 1 -- 172.21.15.162:0/1632527596 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe49c03f070 con 0x7fe4ac1011e0 2024-09-06T14:20:25.291 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:25.284+0000 7fe4b20b7640 1 -- 172.21.15.162:0/1632527596 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4ac19e200 con 0x7fe4ac1011e0 2024-09-06T14:20:25.291 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:25.284+0000 7fe4a8ff9640 1 -- 172.21.15.162:0/1632527596 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7fe49c003a70 con 0x7fe4ac1011e0 2024-09-06T14:20:25.291 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:25.284+0000 7fe4a8ff9640 1 -- 172.21.15.162:0/1632527596 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe49c035030 con 0x7fe4ac1011e0 2024-09-06T14:20:25.291 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:25.284+0000 7fe4b20b7640 1 -- 172.21.15.162:0/1632527596 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4ac1a3310 con 0x7fe4ac1011e0 2024-09-06T14:20:25.292 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:25.285+0000 7fe4a8ff9640 1 -- 172.21.15.162:0/1632527596 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 13) ==== 50132+0+0 (secure 0 0 0) 0x7fe49c042070 con 0x7fe4ac1011e0 2024-09-06T14:20:25.292 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:25.285+0000 7fe4b20b7640 1 -- 172.21.15.162:0/1632527596 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4ac06b2a0 con 0x7fe4ac1011e0 2024-09-06T14:20:25.292 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:25.285+0000 7fe4a8ff9640 1 --2- 172.21.15.162:0/1632527596 >> v2:172.21.15.142:6800/853729507 conn(0x7fe49403daf0 0x7fe49403ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:20:25.292 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:25.285+0000 7fe4a8ff9640 1 -- 172.21.15.162:0/1632527596 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 983+0+0 (secure 0 0 0) 0x7fe49c072900 con 0x7fe4ac1011e0 2024-09-06T14:20:25.292 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:25.285+0000 7fe4aaffd640 1 --2- 172.21.15.162:0/1632527596 >> v2:172.21.15.142:6800/853729507 conn(0x7fe49403daf0 0x7fe49403ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:20:25.292 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:25.286+0000 7fe4aaffd640 1 --2- 172.21.15.162:0/1632527596 >> v2:172.21.15.142:6800/853729507 conn(0x7fe49403daf0 0x7fe49403ffb0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fe4a00098c0 tx=0x7fe4a0006db0 comp rx=0 tx=0).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:20:25.292 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:25.290+0000 7fe4a8ff9640 1 -- 172.21.15.162:0/1632527596 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7fe49c039df0 con 0x7fe4ac1011e0 2024-09-06T14:20:25.506 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:25.508+0000 7fe4b20b7640 1 -- 172.21.15.162:0/1632527596 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7fe4ac101640 con 0x7fe4ac1011e0 2024-09-06T14:20:25.506 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:25.508+0000 7fe4a8ff9640 1 -- 172.21.15.162:0/1632527596 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+749 (secure 0 0 0) 0x7fe49c003d00 con 0x7fe4ac1011e0 2024-09-06T14:20:25.506 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-06T14:20:25.506 INFO:teuthology.orchestra.run.smithi162.stdout:{"epoch":1,"fsid":"ce32462e-6c5a-11ef-bce4-c7b262605968","modified":"2024-09-06T14:19:01.768092Z","created":"2024-09-06T14:19:01.768092Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-06T14:20:25.507 INFO:teuthology.orchestra.run.smithi162.stderr:dumped monmap epoch 1 2024-09-06T14:20:25.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:25.511+0000 7fe4b20b7640 1 -- 172.21.15.162:0/1632527596 >> v2:172.21.15.142:6800/853729507 conn(0x7fe49403daf0 msgr2=0x7fe49403ffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:20:25.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:25.511+0000 7fe4b20b7640 1 --2- 172.21.15.162:0/1632527596 >> v2:172.21.15.142:6800/853729507 conn(0x7fe49403daf0 0x7fe49403ffb0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fe4a00098c0 tx=0x7fe4a0006db0 comp rx=0 tx=0).stop 2024-09-06T14:20:25.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:25.511+0000 7fe4b20b7640 1 -- 172.21.15.162:0/1632527596 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe4ac1011e0 msgr2=0x7fe4ac19da30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:20:25.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:25.511+0000 7fe4b20b7640 1 --2- 172.21.15.162:0/1632527596 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe4ac1011e0 0x7fe4ac19da30 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fe49c005ab0 tx=0x7fe49c0048f0 comp rx=0 tx=0).stop 2024-09-06T14:20:25.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:25.511+0000 7fe4b20b7640 1 -- 172.21.15.162:0/1632527596 shutdown_connections 2024-09-06T14:20:25.508 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:25.511+0000 7fe4b20b7640 1 --2- 172.21.15.162:0/1632527596 >> v2:172.21.15.142:6800/853729507 conn(0x7fe49403daf0 0x7fe49403ffb0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:20:25.508 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:25.511+0000 7fe4b20b7640 1 --2- 172.21.15.162:0/1632527596 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe4ac1011e0 0x7fe4ac19da30 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:20:25.508 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:25.511+0000 7fe4b20b7640 1 -- 172.21.15.162:0/1632527596 >> 172.21.15.162:0/1632527596 conn(0x7fe4ac0fce90 msgr2=0x7fe4ac0fda90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:20:25.508 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:25.511+0000 7fe4b20b7640 1 -- 172.21.15.162:0/1632527596 shutdown_connections 2024-09-06T14:20:25.508 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:25.511+0000 7fe4b20b7640 1 -- 172.21.15.162:0/1632527596 wait complete. 2024-09-06T14:20:25.957 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:25 smithi142 ceph-mon[29958]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-06T14:20:25.957 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:25 smithi142 ceph-mon[29958]: from='client.? 172.21.15.162:0/1632527596' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-06T14:20:27.123 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-06T14:20:27.124 DEBUG:teuthology.orchestra.run.smithi162:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph mon dump -f json 2024-09-06T14:20:27.539 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.b/config 2024-09-06T14:20:27.997 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:27.992+0000 7f3c8bc15640 1 -- 172.21.15.162:0/2567734494 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3c8406a6e0 msgr2=0x7f3c8406aac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:20:27.997 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:27.992+0000 7f3c8bc15640 1 --2- 172.21.15.162:0/2567734494 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3c8406a6e0 0x7f3c8406aac0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f3c74009c20 tx=0x7f3c7402aa90 comp rx=0 tx=0).stop 2024-09-06T14:20:27.997 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:27.992+0000 7f3c8bc15640 1 -- 172.21.15.162:0/2567734494 shutdown_connections 2024-09-06T14:20:27.997 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:27.992+0000 7f3c8bc15640 1 --2- 172.21.15.162:0/2567734494 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3c8406a6e0 0x7f3c8406aac0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:20:27.997 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:27.992+0000 7f3c8bc15640 1 -- 172.21.15.162:0/2567734494 >> 172.21.15.162:0/2567734494 conn(0x7f3c84076ed0 msgr2=0x7f3c840772e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:20:27.997 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:27.992+0000 7f3c8bc15640 1 -- 172.21.15.162:0/2567734494 shutdown_connections 2024-09-06T14:20:27.998 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:27.992+0000 7f3c8bc15640 1 -- 172.21.15.162:0/2567734494 wait complete. 2024-09-06T14:20:27.998 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:27.993+0000 7f3c8bc15640 1 Processor -- start 2024-09-06T14:20:27.998 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:27.993+0000 7f3c8bc15640 1 -- start start 2024-09-06T14:20:27.998 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:27.993+0000 7f3c8bc15640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3c8406a6e0 0x7f3c84197570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:20:27.998 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:27.993+0000 7f3c8bc15640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f3c8410bf70 con 0x7f3c8406a6e0 2024-09-06T14:20:27.998 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:27.993+0000 7f3c899b1640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3c8406a6e0 0x7f3c84197570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:20:27.998 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:27.993+0000 7f3c899b1640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3c8406a6e0 0x7f3c84197570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.162:44120/0 (socket says 172.21.15.162:44120) 2024-09-06T14:20:27.998 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:27.993+0000 7f3c899b1640 1 -- 172.21.15.162:0/1251782708 learned_addr learned my addr 172.21.15.162:0/1251782708 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-09-06T14:20:27.998 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:27.994+0000 7f3c899b1640 1 -- 172.21.15.162:0/1251782708 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c84197ab0 con 0x7f3c8406a6e0 2024-09-06T14:20:27.998 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:27.994+0000 7f3c899b1640 1 --2- 172.21.15.162:0/1251782708 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3c8406a6e0 0x7f3c84197570 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3c74009d40 tx=0x7f3c7402e960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:20:27.998 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:27.994+0000 7f3c82ffd640 1 -- 172.21.15.162:0/1251782708 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3c74047070 con 0x7f3c8406a6e0 2024-09-06T14:20:27.998 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:27.994+0000 7f3c82ffd640 1 -- 172.21.15.162:0/1251782708 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7f3c74003df0 con 0x7f3c8406a6e0 2024-09-06T14:20:27.998 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:27.994+0000 7f3c82ffd640 1 -- 172.21.15.162:0/1251782708 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3c74033050 con 0x7f3c8406a6e0 2024-09-06T14:20:27.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:27.994+0000 7f3c8bc15640 1 -- 172.21.15.162:0/1251782708 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c84197d40 con 0x7f3c8406a6e0 2024-09-06T14:20:27.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:27.994+0000 7f3c8bc15640 1 -- 172.21.15.162:0/1251782708 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c8419ce50 con 0x7f3c8406a6e0 2024-09-06T14:20:27.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:27.995+0000 7f3c8bc15640 1 -- 172.21.15.162:0/1251782708 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c841038e0 con 0x7f3c8406a6e0 2024-09-06T14:20:27.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:27.995+0000 7f3c82ffd640 1 -- 172.21.15.162:0/1251782708 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 13) ==== 50132+0+0 (secure 0 0 0) 0x7f3c7404a070 con 0x7f3c8406a6e0 2024-09-06T14:20:27.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:27.995+0000 7f3c82ffd640 1 --2- 172.21.15.162:0/1251782708 >> v2:172.21.15.142:6800/853729507 conn(0x7f3c6c03daf0 0x7f3c6c03ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:20:27.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:27.995+0000 7f3c82ffd640 1 -- 172.21.15.162:0/1251782708 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 983+0+0 (secure 0 0 0) 0x7f3c7403e030 con 0x7f3c8406a6e0 2024-09-06T14:20:27.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:27.996+0000 7f3c891b0640 1 --2- 172.21.15.162:0/1251782708 >> v2:172.21.15.142:6800/853729507 conn(0x7f3c6c03daf0 0x7f3c6c03ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:20:27.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:27.996+0000 7f3c891b0640 1 --2- 172.21.15.162:0/1251782708 >> v2:172.21.15.142:6800/853729507 conn(0x7f3c6c03daf0 0x7f3c6c03ffb0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f3c780098c0 tx=0x7f3c78006db0 comp rx=0 tx=0).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:20:27.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:28.000+0000 7f3c82ffd640 1 -- 172.21.15.162:0/1251782708 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7f3c7403f2d0 con 0x7f3c8406a6e0 2024-09-06T14:20:28.207 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:27 smithi142 ceph-mon[29958]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-06T14:20:28.220 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:28.224+0000 7f3c8bc15640 1 -- 172.21.15.162:0/1251782708 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f3c8406ab40 con 0x7f3c8406a6e0 2024-09-06T14:20:28.220 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:28.225+0000 7f3c82ffd640 1 -- 172.21.15.162:0/1251782708 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+749 (secure 0 0 0) 0x7f3c74004770 con 0x7f3c8406a6e0 2024-09-06T14:20:28.220 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-06T14:20:28.221 INFO:teuthology.orchestra.run.smithi162.stdout:{"epoch":1,"fsid":"ce32462e-6c5a-11ef-bce4-c7b262605968","modified":"2024-09-06T14:19:01.768092Z","created":"2024-09-06T14:19:01.768092Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-06T14:20:28.221 INFO:teuthology.orchestra.run.smithi162.stderr:dumped monmap epoch 1 2024-09-06T14:20:28.222 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:28.226+0000 7f3c8bc15640 1 -- 172.21.15.162:0/1251782708 >> v2:172.21.15.142:6800/853729507 conn(0x7f3c6c03daf0 msgr2=0x7f3c6c03ffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:20:28.222 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:28.226+0000 7f3c8bc15640 1 --2- 172.21.15.162:0/1251782708 >> v2:172.21.15.142:6800/853729507 conn(0x7f3c6c03daf0 0x7f3c6c03ffb0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f3c780098c0 tx=0x7f3c78006db0 comp rx=0 tx=0).stop 2024-09-06T14:20:28.222 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:28.227+0000 7f3c8bc15640 1 -- 172.21.15.162:0/1251782708 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3c8406a6e0 msgr2=0x7f3c84197570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:20:28.223 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:28.227+0000 7f3c8bc15640 1 --2- 172.21.15.162:0/1251782708 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3c8406a6e0 0x7f3c84197570 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3c74009d40 tx=0x7f3c7402e960 comp rx=0 tx=0).stop 2024-09-06T14:20:28.223 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:28.227+0000 7f3c8bc15640 1 -- 172.21.15.162:0/1251782708 shutdown_connections 2024-09-06T14:20:28.223 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:28.227+0000 7f3c8bc15640 1 --2- 172.21.15.162:0/1251782708 >> v2:172.21.15.142:6800/853729507 conn(0x7f3c6c03daf0 0x7f3c6c03ffb0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:20:28.223 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:28.227+0000 7f3c8bc15640 1 --2- 172.21.15.162:0/1251782708 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3c8406a6e0 0x7f3c84197570 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:20:28.223 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:28.227+0000 7f3c8bc15640 1 -- 172.21.15.162:0/1251782708 >> 172.21.15.162:0/1251782708 conn(0x7f3c84076ed0 msgr2=0x7f3c84108700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:20:28.223 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:28.227+0000 7f3c8bc15640 1 -- 172.21.15.162:0/1251782708 shutdown_connections 2024-09-06T14:20:28.223 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:28.227+0000 7f3c8bc15640 1 -- 172.21.15.162:0/1251782708 wait complete. 2024-09-06T14:20:29.207 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:28 smithi142 ceph-mon[29958]: from='client.? 172.21.15.162:0/1251782708' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-06T14:20:29.295 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:29 smithi162 systemd[1]: Started Ceph mon.b for ce32462e-6c5a-11ef-bce4-c7b262605968. 2024-09-06T14:20:29.898 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-06T14:20:29.898 DEBUG:teuthology.orchestra.run.smithi162:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph mon dump -f json 2024-09-06T14:20:30.156 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.b/config 2024-09-06T14:20:34.190 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:34.194+0000 7fcd3ee35640 1 -- 172.21.15.162:0/1859867639 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcd20003530 msgr2=0x7fcd200059c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:20:34.190 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:34.194+0000 7fcd3ee35640 1 --2- 172.21.15.162:0/1859867639 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcd20003530 0x7fcd200059c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fcd38101260 tx=0x7fcd2802c3f0 comp rx=0 tx=0).stop 2024-09-06T14:20:34.190 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:34.194+0000 7fcd3ee35640 1 -- 172.21.15.162:0/1859867639 shutdown_connections 2024-09-06T14:20:34.190 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:34.194+0000 7fcd3ee35640 1 --2- 172.21.15.162:0/1859867639 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcd20003530 0x7fcd200059c0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:20:34.190 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:34.194+0000 7fcd3ee35640 1 -- 172.21.15.162:0/1859867639 >> 172.21.15.162:0/1859867639 conn(0x7fcd380fcf80 msgr2=0x7fcd380ff3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:20:34.190 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:34.194+0000 7fcd3ee35640 1 -- 172.21.15.162:0/1859867639 shutdown_connections 2024-09-06T14:20:34.190 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:34.194+0000 7fcd3ee35640 1 -- 172.21.15.162:0/1859867639 wait complete. 2024-09-06T14:20:34.192 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:34.194+0000 7fcd3ee35640 1 Processor -- start 2024-09-06T14:20:34.192 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:34.195+0000 7fcd3ee35640 1 -- start start 2024-09-06T14:20:34.192 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:34.195+0000 7fcd3ee35640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcd38197760 0x7fcd38197b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:20:34.193 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:34.195+0000 7fcd3ee35640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcd38198080 0x7fcd3819d040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:20:34.193 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:34.195+0000 7fcd3ee35640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fcd38108920 con 0x7fcd38197760 2024-09-06T14:20:34.193 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:34.195+0000 7fcd3ee35640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fcd381087a0 con 0x7fcd38198080 2024-09-06T14:20:34.193 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:34.195+0000 7fcd37fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcd38198080 0x7fcd3819d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:20:34.193 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:34.195+0000 7fcd3cbd1640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcd38197760 0x7fcd38197b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:20:34.193 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:34.195+0000 7fcd37fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcd38198080 0x7fcd3819d040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:55666/0 (socket says 172.21.15.162:55666) 2024-09-06T14:20:34.193 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:34.195+0000 7fcd37fff640 1 -- 172.21.15.162:0/585470434 learned_addr learned my addr 172.21.15.162:0/585470434 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-09-06T14:20:34.193 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:34.195+0000 7fcd37fff640 1 -- 172.21.15.162:0/585470434 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcd38198080 msgr2=0x7fcd3819d040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-06T14:20:34.193 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:34.195+0000 7fcd37fff640 1 -- 172.21.15.162:0/585470434 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcd38198080 msgr2=0x7fcd3819d040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-06T14:20:34.193 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:34.195+0000 7fcd37fff640 1 --2- 172.21.15.162:0/585470434 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcd38198080 0x7fcd3819d040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-06T14:20:34.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:34.195+0000 7fcd37fff640 1 --2- 172.21.15.162:0/585470434 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcd38198080 0x7fcd3819d040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-06T14:20:34.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:34.195+0000 7fcd3cbd1640 1 -- 172.21.15.162:0/585470434 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcd38198080 msgr2=0x7fcd3819d040 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-06T14:20:34.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:34.195+0000 7fcd3cbd1640 1 --2- 172.21.15.162:0/585470434 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcd38198080 0x7fcd3819d040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:20:34.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:34.195+0000 7fcd3cbd1640 1 -- 172.21.15.162:0/585470434 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd3819d6b0 con 0x7fcd38197760 2024-09-06T14:20:34.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:34.196+0000 7fcd3cbd1640 1 --2- 172.21.15.162:0/585470434 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcd38197760 0x7fcd38197b40 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fcd28013000 tx=0x7fcd280129f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:20:34.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:34.196+0000 7fcd35ffb640 1 -- 172.21.15.162:0/585470434 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7fcd28005050 con 0x7fcd38197760 2024-09-06T14:20:34.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:34.196+0000 7fcd35ffb640 1 -- 172.21.15.162:0/585470434 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7fcd28004620 con 0x7fcd38197760 2024-09-06T14:20:34.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:34.196+0000 7fcd3ee35640 1 -- 172.21.15.162:0/585470434 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd3819d940 con 0x7fcd38197760 2024-09-06T14:20:34.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:34.196+0000 7fcd35ffb640 1 -- 172.21.15.162:0/585470434 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7fcd2803d620 con 0x7fcd38197760 2024-09-06T14:20:34.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:34.196+0000 7fcd3ee35640 1 -- 172.21.15.162:0/585470434 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd381a21b0 con 0x7fcd38197760 2024-09-06T14:20:34.195 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:34.197+0000 7fcd35ffb640 1 -- 172.21.15.162:0/585470434 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 13) ==== 50132+0+0 (secure 0 0 0) 0x7fcd28044020 con 0x7fcd38197760 2024-09-06T14:20:34.195 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:34.197+0000 7fcd3ee35640 1 -- 172.21.15.162:0/585470434 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd381027a0 con 0x7fcd38197760 2024-09-06T14:20:34.195 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:34.198+0000 7fcd35ffb640 1 --2- 172.21.15.162:0/585470434 >> v2:172.21.15.142:6800/853729507 conn(0x7fcd2403d8e0 0x7fcd2403fda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:20:34.195 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:34.198+0000 7fcd35ffb640 1 -- 172.21.15.162:0/585470434 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 983+0+0 (secure 0 0 0) 0x7fcd28073ff0 con 0x7fcd38197760 2024-09-06T14:20:34.196 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:34.198+0000 7fcd37fff640 1 --2- 172.21.15.162:0/585470434 >> v2:172.21.15.142:6800/853729507 conn(0x7fcd2403d8e0 0x7fcd2403fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:20:34.196 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:34.198+0000 7fcd37fff640 1 --2- 172.21.15.162:0/585470434 >> v2:172.21.15.142:6800/853729507 conn(0x7fcd2403d8e0 0x7fcd2403fda0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fcd2c009100 tx=0x7fcd2c008040 comp rx=0 tx=0).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:20:34.198 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:34.202+0000 7fcd35ffb640 1 -- 172.21.15.162:0/585470434 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7fcd2804b020 con 0x7fcd38197760 2024-09-06T14:20:34.409 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:34.413+0000 7fcd3ee35640 1 -- 172.21.15.162:0/585470434 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7fcd38190ee0 con 0x7fcd38197760 2024-09-06T14:20:34.410 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:34.414+0000 7fcd35ffb640 1 -- 172.21.15.162:0/585470434 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1019 (secure 0 0 0) 0x7fcd2804ee20 con 0x7fcd38197760 2024-09-06T14:20:34.410 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-06T14:20:34.410 INFO:teuthology.orchestra.run.smithi162.stdout:{"epoch":2,"fsid":"ce32462e-6c5a-11ef-bce4-c7b262605968","modified":"2024-09-06T14:20:29.067692Z","created":"2024-09-06T14:19:01.768092Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:3300","nonce":0},{"type":"v1","addr":"172.21.15.162:6789","nonce":0}]},"addr":"172.21.15.162:6789/0","public_addr":"172.21.15.162:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-06T14:20:34.410 INFO:teuthology.orchestra.run.smithi162.stderr:dumped monmap epoch 2 2024-09-06T14:20:34.411 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:34 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-06T14:20:34.411 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:34 smithi162 ceph-mon[33787]: mon.a calling monitor election 2024-09-06T14:20:34.411 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:34 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-06T14:20:34.411 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:34 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-06T14:20:34.411 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:34 smithi162 ceph-mon[33787]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-06T14:20:34.411 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:34 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-06T14:20:34.411 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:34 smithi162 ceph-mon[33787]: mon.b calling monitor election 2024-09-06T14:20:34.411 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:34 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-06T14:20:34.411 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:34 smithi162 ceph-mon[33787]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-06T14:20:34.411 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:34 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-06T14:20:34.411 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:34 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-06T14:20:34.412 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:34 smithi162 ceph-mon[33787]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-06T14:20:34.412 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:34 smithi162 ceph-mon[33787]: monmap epoch 2 2024-09-06T14:20:34.412 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:34 smithi162 ceph-mon[33787]: fsid ce32462e-6c5a-11ef-bce4-c7b262605968 2024-09-06T14:20:34.412 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:34 smithi162 ceph-mon[33787]: last_changed 2024-09-06T14:20:29.067692+0000 2024-09-06T14:20:34.412 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:34 smithi162 ceph-mon[33787]: created 2024-09-06T14:19:01.768092+0000 2024-09-06T14:20:34.412 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:34 smithi162 ceph-mon[33787]: min_mon_release 19 (squid) 2024-09-06T14:20:34.412 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:34 smithi162 ceph-mon[33787]: election_strategy: 1 2024-09-06T14:20:34.412 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:34 smithi162 ceph-mon[33787]: 0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-06T14:20:34.412 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:34 smithi162 ceph-mon[33787]: 1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-09-06T14:20:34.412 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:34 smithi162 ceph-mon[33787]: fsmap 2024-09-06T14:20:34.413 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:34 smithi162 ceph-mon[33787]: osdmap e4: 0 total, 0 up, 0 in 2024-09-06T14:20:34.413 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:34 smithi162 ceph-mon[33787]: mgrmap e13: y(active, since 35s) 2024-09-06T14:20:34.413 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:34 smithi162 ceph-mon[33787]: Health detail: HEALTH_WARN Module 'volumes' has failed dependency: No module named 'ceph.fs' 2024-09-06T14:20:34.413 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:34 smithi162 ceph-mon[33787]: [WRN] MGR_MODULE_DEPENDENCY: Module 'volumes' has failed dependency: No module named 'ceph.fs' 2024-09-06T14:20:34.413 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:34 smithi162 ceph-mon[33787]: Module 'volumes' has failed dependency: No module named 'ceph.fs' 2024-09-06T14:20:34.413 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:34 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:34.413 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:34 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:34.413 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:34 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:34.413 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:34 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-06T14:20:34.413 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:34 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:20:34.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:34.416+0000 7fcd3ee35640 1 -- 172.21.15.162:0/585470434 >> v2:172.21.15.142:6800/853729507 conn(0x7fcd2403d8e0 msgr2=0x7fcd2403fda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:20:34.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:34.416+0000 7fcd3ee35640 1 --2- 172.21.15.162:0/585470434 >> v2:172.21.15.142:6800/853729507 conn(0x7fcd2403d8e0 0x7fcd2403fda0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fcd2c009100 tx=0x7fcd2c008040 comp rx=0 tx=0).stop 2024-09-06T14:20:34.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:34.416+0000 7fcd3ee35640 1 -- 172.21.15.162:0/585470434 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcd38197760 msgr2=0x7fcd38197b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:20:34.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:34.416+0000 7fcd3ee35640 1 --2- 172.21.15.162:0/585470434 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcd38197760 0x7fcd38197b40 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fcd28013000 tx=0x7fcd280129f0 comp rx=0 tx=0).stop 2024-09-06T14:20:34.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:34.416+0000 7fcd3ee35640 1 -- 172.21.15.162:0/585470434 shutdown_connections 2024-09-06T14:20:34.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:34.416+0000 7fcd3ee35640 1 --2- 172.21.15.162:0/585470434 >> v2:172.21.15.142:6800/853729507 conn(0x7fcd2403d8e0 0x7fcd2403fda0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:20:34.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:34.416+0000 7fcd3ee35640 1 --2- 172.21.15.162:0/585470434 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcd38198080 0x7fcd3819d040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:20:34.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:34.416+0000 7fcd3ee35640 1 --2- 172.21.15.162:0/585470434 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcd38197760 0x7fcd38197b40 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:20:34.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:34.416+0000 7fcd3ee35640 1 -- 172.21.15.162:0/585470434 >> 172.21.15.162:0/585470434 conn(0x7fcd380fcf80 msgr2=0x7fcd3806b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:20:34.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:34.416+0000 7fcd3ee35640 1 -- 172.21.15.162:0/585470434 shutdown_connections 2024-09-06T14:20:34.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:34.416+0000 7fcd3ee35640 1 -- 172.21.15.162:0/585470434 wait complete. 2024-09-06T14:20:34.457 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:34 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-06T14:20:34.458 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:34 smithi142 ceph-mon[29958]: mon.a calling monitor election 2024-09-06T14:20:34.458 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:34 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-06T14:20:34.458 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:34 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-06T14:20:34.458 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:34 smithi142 ceph-mon[29958]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-06T14:20:34.458 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:34 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-06T14:20:34.458 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:34 smithi142 ceph-mon[29958]: mon.b calling monitor election 2024-09-06T14:20:34.458 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:34 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-06T14:20:34.458 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:34 smithi142 ceph-mon[29958]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-06T14:20:34.458 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:34 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-06T14:20:34.458 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:34 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-06T14:20:34.458 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:34 smithi142 ceph-mon[29958]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-06T14:20:34.458 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:34 smithi142 ceph-mon[29958]: monmap epoch 2 2024-09-06T14:20:34.459 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:34 smithi142 ceph-mon[29958]: fsid ce32462e-6c5a-11ef-bce4-c7b262605968 2024-09-06T14:20:34.459 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:34 smithi142 ceph-mon[29958]: last_changed 2024-09-06T14:20:29.067692+0000 2024-09-06T14:20:34.459 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:34 smithi142 ceph-mon[29958]: created 2024-09-06T14:19:01.768092+0000 2024-09-06T14:20:34.459 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:34 smithi142 ceph-mon[29958]: min_mon_release 19 (squid) 2024-09-06T14:20:34.459 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:34 smithi142 ceph-mon[29958]: election_strategy: 1 2024-09-06T14:20:34.459 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:34 smithi142 ceph-mon[29958]: 0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-06T14:20:34.459 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:34 smithi142 ceph-mon[29958]: 1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-09-06T14:20:34.459 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:34 smithi142 ceph-mon[29958]: fsmap 2024-09-06T14:20:34.459 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:34 smithi142 ceph-mon[29958]: osdmap e4: 0 total, 0 up, 0 in 2024-09-06T14:20:34.459 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:34 smithi142 ceph-mon[29958]: mgrmap e13: y(active, since 35s) 2024-09-06T14:20:34.459 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:34 smithi142 ceph-mon[29958]: Health detail: HEALTH_WARN Module 'volumes' has failed dependency: No module named 'ceph.fs' 2024-09-06T14:20:34.459 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:34 smithi142 ceph-mon[29958]: [WRN] MGR_MODULE_DEPENDENCY: Module 'volumes' has failed dependency: No module named 'ceph.fs' 2024-09-06T14:20:34.459 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:34 smithi142 ceph-mon[29958]: Module 'volumes' has failed dependency: No module named 'ceph.fs' 2024-09-06T14:20:34.460 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:34 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:34.460 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:34 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:34.460 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:34 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:34.460 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:34 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-06T14:20:34.460 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:34 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:20:35.295 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:35 smithi162 ceph-mon[33787]: Deploying daemon mon.c on smithi142 2024-09-06T14:20:35.295 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:35 smithi162 ceph-mon[33787]: from='client.? 172.21.15.162:0/585470434' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-06T14:20:35.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:35 smithi162 ceph-mon[33787]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-06T14:20:35.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:35 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-06T14:20:35.457 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:35 smithi142 ceph-mon[29958]: Deploying daemon mon.c on smithi142 2024-09-06T14:20:35.457 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:35 smithi142 ceph-mon[29958]: from='client.? 172.21.15.162:0/585470434' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-06T14:20:35.457 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:35 smithi142 ceph-mon[29958]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-06T14:20:35.457 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:35 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-06T14:20:36.046 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-06T14:20:36.046 DEBUG:teuthology.orchestra.run.smithi162:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph mon dump -f json 2024-09-06T14:20:36.298 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.b/config 2024-09-06T14:20:36.713 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:36.712+0000 7f0be77eb640 1 -- 172.21.15.162:0/813605677 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0be01032b0 msgr2=0x7f0be01007b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:20:36.714 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:36.712+0000 7f0be77eb640 1 --2- 172.21.15.162:0/813605677 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0be01032b0 0x7f0be01007b0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f0bdc009880 tx=0x7f0bdc02a5d0 comp rx=0 tx=0).stop 2024-09-06T14:20:36.714 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:36.712+0000 7f0be77eb640 1 -- 172.21.15.162:0/813605677 shutdown_connections 2024-09-06T14:20:36.714 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:36.712+0000 7f0be77eb640 1 --2- 172.21.15.162:0/813605677 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0be01032b0 0x7f0be01007b0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:20:36.714 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:36.712+0000 7f0be77eb640 1 -- 172.21.15.162:0/813605677 >> 172.21.15.162:0/813605677 conn(0x7f0be00fc720 msgr2=0x7f0be00feb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:20:36.714 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:36.712+0000 7f0be77eb640 1 -- 172.21.15.162:0/813605677 shutdown_connections 2024-09-06T14:20:36.714 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:36.712+0000 7f0be77eb640 1 -- 172.21.15.162:0/813605677 wait complete. 2024-09-06T14:20:36.714 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:36.713+0000 7f0be77eb640 1 Processor -- start 2024-09-06T14:20:36.714 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:36.713+0000 7f0be77eb640 1 -- start start 2024-09-06T14:20:36.714 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:36.713+0000 7f0be77eb640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0be01032b0 0x7f0be0199c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:20:36.714 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:36.713+0000 7f0be77eb640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0be019a1a0 0x7f0be019f540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:20:36.714 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:36.713+0000 7f0be77eb640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f0be010e280 con 0x7f0be01032b0 2024-09-06T14:20:36.715 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:36.713+0000 7f0be77eb640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f0be010e100 con 0x7f0be019a1a0 2024-09-06T14:20:36.715 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:36.713+0000 7f0be5587640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0be01032b0 0x7f0be0199c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:20:36.715 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:36.713+0000 7f0be5587640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0be01032b0 0x7f0be0199c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.162:35074/0 (socket says 172.21.15.162:35074) 2024-09-06T14:20:36.715 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:36.713+0000 7f0be5587640 1 -- 172.21.15.162:0/1164203435 learned_addr learned my addr 172.21.15.162:0/1164203435 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-09-06T14:20:36.715 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:36.714+0000 7f0be4d86640 1 --2- 172.21.15.162:0/1164203435 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0be019a1a0 0x7f0be019f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:20:36.715 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:36.714+0000 7f0be4d86640 1 -- 172.21.15.162:0/1164203435 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0be019a1a0 msgr2=0x7f0be019f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-06T14:20:36.715 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:36.714+0000 7f0be4d86640 1 -- 172.21.15.162:0/1164203435 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0be019a1a0 msgr2=0x7f0be019f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-06T14:20:36.715 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:36.714+0000 7f0be4d86640 1 --2- 172.21.15.162:0/1164203435 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0be019a1a0 0x7f0be019f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-06T14:20:36.715 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:36.714+0000 7f0be4d86640 1 --2- 172.21.15.162:0/1164203435 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0be019a1a0 0x7f0be019f540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-06T14:20:36.715 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:36.714+0000 7f0be5587640 1 -- 172.21.15.162:0/1164203435 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0be019a1a0 msgr2=0x7f0be019f540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-06T14:20:36.715 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:36.714+0000 7f0be5587640 1 --2- 172.21.15.162:0/1164203435 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0be019a1a0 0x7f0be019f540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:20:36.715 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:36.714+0000 7f0be5587640 1 -- 172.21.15.162:0/1164203435 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0be019a820 con 0x7f0be01032b0 2024-09-06T14:20:36.715 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:36.714+0000 7f0be5587640 1 --2- 172.21.15.162:0/1164203435 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0be01032b0 0x7f0be0199c60 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f0bdc005ab0 tx=0x7f0bdc0048f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:20:36.716 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:36.714+0000 7f0bd67fc640 1 -- 172.21.15.162:0/1164203435 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f0bdc035030 con 0x7f0be01032b0 2024-09-06T14:20:36.716 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:36.714+0000 7f0bd67fc640 1 -- 172.21.15.162:0/1164203435 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7f0bdc004cf0 con 0x7f0be01032b0 2024-09-06T14:20:36.716 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:36.714+0000 7f0be77eb640 1 -- 172.21.15.162:0/1164203435 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0be019fae0 con 0x7f0be01032b0 2024-09-06T14:20:36.716 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:36.714+0000 7f0bd67fc640 1 -- 172.21.15.162:0/1164203435 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f0bdc03b700 con 0x7f0be01032b0 2024-09-06T14:20:36.716 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:36.714+0000 7f0be77eb640 1 -- 172.21.15.162:0/1164203435 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0be01a00a0 con 0x7f0be01032b0 2024-09-06T14:20:36.716 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:36.715+0000 7f0be77eb640 1 -- 172.21.15.162:0/1164203435 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0be01017c0 con 0x7f0be01032b0 2024-09-06T14:20:36.716 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:36.715+0000 7f0bd67fc640 1 -- 172.21.15.162:0/1164203435 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 13) ==== 50132+0+0 (secure 0 0 0) 0x7f0bdc042070 con 0x7f0be01032b0 2024-09-06T14:20:36.716 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:36.715+0000 7f0bd67fc640 1 --2- 172.21.15.162:0/1164203435 >> v2:172.21.15.142:6800/853729507 conn(0x7f0bcc03d8d0 0x7f0bcc03fd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:20:36.716 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:36.715+0000 7f0bd67fc640 1 -- 172.21.15.162:0/1164203435 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 983+0+0 (secure 0 0 0) 0x7f0bdc0727a0 con 0x7f0be01032b0 2024-09-06T14:20:36.716 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:36.716+0000 7f0be4d86640 1 --2- 172.21.15.162:0/1164203435 >> v2:172.21.15.142:6800/853729507 conn(0x7f0bcc03d8d0 0x7f0bcc03fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:20:36.716 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:36.716+0000 7f0be4d86640 1 --2- 172.21.15.162:0/1164203435 >> v2:172.21.15.142:6800/853729507 conn(0x7f0bcc03d8d0 0x7f0bcc03fd90 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0bd0006fd0 tx=0x7f0bd0008040 comp rx=0 tx=0).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:20:36.717 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:36.720+0000 7f0bd67fc640 1 -- 172.21.15.162:0/1164203435 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7f0bdc036260 con 0x7f0be01032b0 2024-09-06T14:20:36.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:36.949+0000 7f0be77eb640 1 -- 172.21.15.162:0/1164203435 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f0be0100830 con 0x7f0be01032b0 2024-09-06T14:20:36.946 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:36.950+0000 7f0bd67fc640 1 -- 172.21.15.162:0/1164203435 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1019 (secure 0 0 0) 0x7f0bdc037dc0 con 0x7f0be01032b0 2024-09-06T14:20:36.946 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-06T14:20:36.946 INFO:teuthology.orchestra.run.smithi162.stdout:{"epoch":2,"fsid":"ce32462e-6c5a-11ef-bce4-c7b262605968","modified":"2024-09-06T14:20:29.067692Z","created":"2024-09-06T14:19:01.768092Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:3300","nonce":0},{"type":"v1","addr":"172.21.15.162:6789","nonce":0}]},"addr":"172.21.15.162:6789/0","public_addr":"172.21.15.162:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-06T14:20:36.947 INFO:teuthology.orchestra.run.smithi162.stderr:dumped monmap epoch 2 2024-09-06T14:20:36.948 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:36.952+0000 7f0be77eb640 1 -- 172.21.15.162:0/1164203435 >> v2:172.21.15.142:6800/853729507 conn(0x7f0bcc03d8d0 msgr2=0x7f0bcc03fd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:20:36.948 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:36.952+0000 7f0be77eb640 1 --2- 172.21.15.162:0/1164203435 >> v2:172.21.15.142:6800/853729507 conn(0x7f0bcc03d8d0 0x7f0bcc03fd90 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0bd0006fd0 tx=0x7f0bd0008040 comp rx=0 tx=0).stop 2024-09-06T14:20:36.948 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:36.952+0000 7f0be77eb640 1 -- 172.21.15.162:0/1164203435 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0be01032b0 msgr2=0x7f0be0199c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:20:36.948 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:36.952+0000 7f0be77eb640 1 --2- 172.21.15.162:0/1164203435 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0be01032b0 0x7f0be0199c60 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f0bdc005ab0 tx=0x7f0bdc0048f0 comp rx=0 tx=0).stop 2024-09-06T14:20:36.948 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:36.952+0000 7f0be77eb640 1 -- 172.21.15.162:0/1164203435 shutdown_connections 2024-09-06T14:20:36.949 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:36.952+0000 7f0be77eb640 1 --2- 172.21.15.162:0/1164203435 >> v2:172.21.15.142:6800/853729507 conn(0x7f0bcc03d8d0 0x7f0bcc03fd90 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:20:36.949 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:36.952+0000 7f0be77eb640 1 --2- 172.21.15.162:0/1164203435 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0be019a1a0 0x7f0be019f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:20:36.949 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:36.952+0000 7f0be77eb640 1 --2- 172.21.15.162:0/1164203435 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0be01032b0 0x7f0be0199c60 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:20:36.949 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:36.952+0000 7f0be77eb640 1 -- 172.21.15.162:0/1164203435 >> 172.21.15.162:0/1164203435 conn(0x7f0be00fc720 msgr2=0x7f0be0109ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:20:36.949 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:36.952+0000 7f0be77eb640 1 -- 172.21.15.162:0/1164203435 shutdown_connections 2024-09-06T14:20:36.949 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:36.952+0000 7f0be77eb640 1 -- 172.21.15.162:0/1164203435 wait complete. 2024-09-06T14:20:37.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:36 smithi142 systemd[1]: Starting Ceph mon.c for ce32462e-6c5a-11ef-bce4-c7b262605968... 2024-09-06T14:20:37.421 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 podman[37353]: 2024-09-06 14:20:37.066986911 +0000 UTC m=+0.016064009 image pull 1c45aca0610a312233aa2613f7b416b2884ae14779f1297041485ab28f1c8110 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c86edd48571ab22373283621eb9feebce9df9c291d4164cfc4df1a09493e1f30 2024-09-06T14:20:37.422 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 podman[37353]: 2024-09-06 14:20:37.175109562 +0000 UTC m=+0.124186646 container create 991d8632ae3f036d4a3b7ad111243e1bda9dd55176b080f99983007d32f94452 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c86edd48571ab22373283621eb9feebce9df9c291d4164cfc4df1a09493e1f30, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mon-c, RELEASE=wip-athakkar-testing-2024-09-04-1837-8b38c33, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS) 2024-09-06T14:20:37.422 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 podman[37353]: 2024-09-06 14:20:37.402391646 +0000 UTC m=+0.351468730 container init 991d8632ae3f036d4a3b7ad111243e1bda9dd55176b080f99983007d32f94452 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c86edd48571ab22373283621eb9feebce9df9c291d4164cfc4df1a09493e1f30, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mon-c, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, RELEASE=wip-athakkar-testing-2024-09-04-1837-8b38c33, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, GIT_BRANCH=HEAD) 2024-09-06T14:20:37.422 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 podman[37353]: 2024-09-06 14:20:37.405454027 +0000 UTC m=+0.354531114 container start 991d8632ae3f036d4a3b7ad111243e1bda9dd55176b080f99983007d32f94452 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c86edd48571ab22373283621eb9feebce9df9c291d4164cfc4df1a09493e1f30, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mon-c, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-athakkar-testing-2024-09-04-1837-8b38c33, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-09-06T14:20:37.707 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: set uid:gid to 167:167 (ceph:ceph) 2024-09-06T14:20:37.708 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: ceph version 19.3.0-4704-g8b38c33c (8b38c33cecf4b0b7a9b3116eb9089e21e444ac73) squid (dev), process ceph-mon, pid 2 2024-09-06T14:20:37.708 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: pidfile_write: ignore empty --pid-file 2024-09-06T14:20:37.708 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: load: jerasure load: lrc 2024-09-06T14:20:37.708 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: RocksDB version: 7.9.2 2024-09-06T14:20:37.708 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Git sha 0 2024-09-06T14:20:37.708 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Compile date 2024-09-04 13:52:53 2024-09-06T14:20:37.708 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: DB SUMMARY 2024-09-06T14:20:37.708 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: DB Session ID: FVN8X5F1A64HTU033PD1 2024-09-06T14:20:37.708 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: CURRENT file: CURRENT 2024-09-06T14:20:37.708 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: IDENTITY file: IDENTITY 2024-09-06T14:20:37.708 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-09-06T14:20:37.709 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-09-06T14:20:37.709 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 636 ; 2024-09-06T14:20:37.709 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.error_if_exists: 0 2024-09-06T14:20:37.709 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.create_if_missing: 0 2024-09-06T14:20:37.709 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.paranoid_checks: 1 2024-09-06T14:20:37.709 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-06T14:20:37.709 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-06T14:20:37.709 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-06T14:20:37.709 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.env: 0x55b719a80160 2024-09-06T14:20:37.709 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.fs: PosixFileSystem 2024-09-06T14:20:37.709 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.info_log: 0x55b71b763880 2024-09-06T14:20:37.709 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.max_file_opening_threads: 16 2024-09-06T14:20:37.709 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.statistics: (nil) 2024-09-06T14:20:37.709 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.use_fsync: 0 2024-09-06T14:20:37.710 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.max_log_file_size: 0 2024-09-06T14:20:37.710 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-06T14:20:37.710 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-06T14:20:37.710 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.keep_log_file_num: 1000 2024-09-06T14:20:37.710 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.recycle_log_file_num: 0 2024-09-06T14:20:37.710 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.allow_fallocate: 1 2024-09-06T14:20:37.710 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.allow_mmap_reads: 0 2024-09-06T14:20:37.710 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.allow_mmap_writes: 0 2024-09-06T14:20:37.710 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.use_direct_reads: 0 2024-09-06T14:20:37.710 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-06T14:20:37.710 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.create_missing_column_families: 0 2024-09-06T14:20:37.710 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.db_log_dir: 2024-09-06T14:20:37.710 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.wal_dir: 2024-09-06T14:20:37.710 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-06T14:20:37.712 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-06T14:20:37.712 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-06T14:20:37.712 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-06T14:20:37.712 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-06T14:20:37.712 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-06T14:20:37.712 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.advise_random_on_open: 1 2024-09-06T14:20:37.713 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.db_write_buffer_size: 0 2024-09-06T14:20:37.713 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.write_buffer_manager: 0x55b71b766e60 2024-09-06T14:20:37.713 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-06T14:20:37.713 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-06T14:20:37.713 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-06T14:20:37.713 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.rate_limiter: (nil) 2024-09-06T14:20:37.713 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-06T14:20:37.713 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.wal_recovery_mode: 2 2024-09-06T14:20:37.713 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.enable_thread_tracking: 0 2024-09-06T14:20:37.713 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.enable_pipelined_write: 0 2024-09-06T14:20:37.713 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.unordered_write: 0 2024-09-06T14:20:37.713 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-06T14:20:37.714 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-06T14:20:37.714 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-06T14:20:37.714 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-06T14:20:37.714 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.row_cache: None 2024-09-06T14:20:37.714 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.wal_filter: None 2024-09-06T14:20:37.714 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-06T14:20:37.714 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.allow_ingest_behind: 0 2024-09-06T14:20:37.714 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.two_write_queues: 0 2024-09-06T14:20:37.714 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.manual_wal_flush: 0 2024-09-06T14:20:37.714 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.wal_compression: 0 2024-09-06T14:20:37.714 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.atomic_flush: 0 2024-09-06T14:20:37.714 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-06T14:20:37.715 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-06T14:20:37.715 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-06T14:20:37.715 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.log_readahead_size: 0 2024-09-06T14:20:37.715 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-06T14:20:37.715 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.best_efforts_recovery: 0 2024-09-06T14:20:37.715 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-06T14:20:37.715 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-06T14:20:37.715 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.allow_data_in_errors: 0 2024-09-06T14:20:37.715 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.db_host_id: __hostname__ 2024-09-06T14:20:37.715 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-06T14:20:37.715 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.max_background_jobs: 2 2024-09-06T14:20:37.715 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.max_background_compactions: -1 2024-09-06T14:20:37.715 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.max_subcompactions: 1 2024-09-06T14:20:37.715 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-06T14:20:37.716 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-06T14:20:37.716 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-06T14:20:37.716 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.max_total_wal_size: 0 2024-09-06T14:20:37.716 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-06T14:20:37.716 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-06T14:20:37.716 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-06T14:20:37.716 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-06T14:20:37.716 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.max_open_files: -1 2024-09-06T14:20:37.716 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.bytes_per_sync: 0 2024-09-06T14:20:37.716 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-06T14:20:37.716 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-06T14:20:37.716 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.compaction_readahead_size: 0 2024-09-06T14:20:37.717 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.max_background_flushes: -1 2024-09-06T14:20:37.717 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Compression algorithms supported: 2024-09-06T14:20:37.717 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: kZSTD supported: 0 2024-09-06T14:20:37.717 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: kXpressCompression supported: 0 2024-09-06T14:20:37.717 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: kBZip2Compression supported: 0 2024-09-06T14:20:37.717 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-06T14:20:37.717 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: kLZ4Compression supported: 1 2024-09-06T14:20:37.717 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: kZlibCompression supported: 1 2024-09-06T14:20:37.718 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: kLZ4HCCompression supported: 1 2024-09-06T14:20:37.718 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: kSnappyCompression supported: 1 2024-09-06T14:20:37.718 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-06T14:20:37.718 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-06T14:20:37.718 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-09-06T14:20:37.718 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-06T14:20:37.718 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-06T14:20:37.718 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.merge_operator: 2024-09-06T14:20:37.718 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.compaction_filter: None 2024-09-06T14:20:37.718 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.compaction_filter_factory: None 2024-09-06T14:20:37.718 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.sst_partitioner_factory: None 2024-09-06T14:20:37.718 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-06T14:20:37.718 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-06T14:20:37.718 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55b71b7634e0) 2024-09-06T14:20:37.719 INFO:journalctl@ceph.mon.c.smithi142.stdout: cache_index_and_filter_blocks: 1 2024-09-06T14:20:37.719 INFO:journalctl@ceph.mon.c.smithi142.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-06T14:20:37.719 INFO:journalctl@ceph.mon.c.smithi142.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-06T14:20:37.719 INFO:journalctl@ceph.mon.c.smithi142.stdout: pin_top_level_index_and_filter: 1 2024-09-06T14:20:37.719 INFO:journalctl@ceph.mon.c.smithi142.stdout: index_type: 0 2024-09-06T14:20:37.719 INFO:journalctl@ceph.mon.c.smithi142.stdout: data_block_index_type: 0 2024-09-06T14:20:37.719 INFO:journalctl@ceph.mon.c.smithi142.stdout: index_shortening: 1 2024-09-06T14:20:37.719 INFO:journalctl@ceph.mon.c.smithi142.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-06T14:20:37.719 INFO:journalctl@ceph.mon.c.smithi142.stdout: checksum: 4 2024-09-06T14:20:37.719 INFO:journalctl@ceph.mon.c.smithi142.stdout: no_block_cache: 0 2024-09-06T14:20:37.719 INFO:journalctl@ceph.mon.c.smithi142.stdout: block_cache: 0x55b71b756850 2024-09-06T14:20:37.719 INFO:journalctl@ceph.mon.c.smithi142.stdout: block_cache_name: BinnedLRUCache 2024-09-06T14:20:37.719 INFO:journalctl@ceph.mon.c.smithi142.stdout: block_cache_options: 2024-09-06T14:20:37.719 INFO:journalctl@ceph.mon.c.smithi142.stdout: capacity : 536870912 2024-09-06T14:20:37.720 INFO:journalctl@ceph.mon.c.smithi142.stdout: num_shard_bits : 4 2024-09-06T14:20:37.720 INFO:journalctl@ceph.mon.c.smithi142.stdout: strict_capacity_limit : 0 2024-09-06T14:20:37.720 INFO:journalctl@ceph.mon.c.smithi142.stdout: high_pri_pool_ratio: 0.000 2024-09-06T14:20:37.720 INFO:journalctl@ceph.mon.c.smithi142.stdout: block_cache_compressed: (nil) 2024-09-06T14:20:37.720 INFO:journalctl@ceph.mon.c.smithi142.stdout: persistent_cache: (nil) 2024-09-06T14:20:37.720 INFO:journalctl@ceph.mon.c.smithi142.stdout: block_size: 4096 2024-09-06T14:20:37.720 INFO:journalctl@ceph.mon.c.smithi142.stdout: block_size_deviation: 10 2024-09-06T14:20:37.720 INFO:journalctl@ceph.mon.c.smithi142.stdout: block_restart_interval: 16 2024-09-06T14:20:37.720 INFO:journalctl@ceph.mon.c.smithi142.stdout: index_block_restart_interval: 1 2024-09-06T14:20:37.720 INFO:journalctl@ceph.mon.c.smithi142.stdout: metadata_block_size: 4096 2024-09-06T14:20:37.720 INFO:journalctl@ceph.mon.c.smithi142.stdout: partition_filters: 0 2024-09-06T14:20:37.720 INFO:journalctl@ceph.mon.c.smithi142.stdout: use_delta_encoding: 1 2024-09-06T14:20:37.720 INFO:journalctl@ceph.mon.c.smithi142.stdout: filter_policy: bloomfilter 2024-09-06T14:20:37.720 INFO:journalctl@ceph.mon.c.smithi142.stdout: whole_key_filtering: 1 2024-09-06T14:20:37.721 INFO:journalctl@ceph.mon.c.smithi142.stdout: verify_compression: 0 2024-09-06T14:20:37.721 INFO:journalctl@ceph.mon.c.smithi142.stdout: read_amp_bytes_per_bit: 0 2024-09-06T14:20:37.721 INFO:journalctl@ceph.mon.c.smithi142.stdout: format_version: 5 2024-09-06T14:20:37.721 INFO:journalctl@ceph.mon.c.smithi142.stdout: enable_index_compression: 1 2024-09-06T14:20:37.721 INFO:journalctl@ceph.mon.c.smithi142.stdout: block_align: 0 2024-09-06T14:20:37.721 INFO:journalctl@ceph.mon.c.smithi142.stdout: max_auto_readahead_size: 262144 2024-09-06T14:20:37.721 INFO:journalctl@ceph.mon.c.smithi142.stdout: prepopulate_block_cache: 0 2024-09-06T14:20:37.721 INFO:journalctl@ceph.mon.c.smithi142.stdout: initial_auto_readahead_size: 8192 2024-09-06T14:20:37.721 INFO:journalctl@ceph.mon.c.smithi142.stdout: num_file_reads_for_auto_readahead: 2 2024-09-06T14:20:37.721 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.write_buffer_size: 33554432 2024-09-06T14:20:37.721 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.max_write_buffer_number: 2 2024-09-06T14:20:37.721 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.compression: NoCompression 2024-09-06T14:20:37.721 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.bottommost_compression: Disabled 2024-09-06T14:20:37.722 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.prefix_extractor: nullptr 2024-09-06T14:20:37.722 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-06T14:20:37.722 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.num_levels: 7 2024-09-06T14:20:37.722 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-06T14:20:37.722 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-06T14:20:37.722 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-06T14:20:37.722 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-06T14:20:37.722 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-06T14:20:37.722 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-06T14:20:37.722 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-06T14:20:37.722 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-06T14:20:37.722 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-06T14:20:37.722 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-06T14:20:37.722 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-06T14:20:37.723 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-06T14:20:37.723 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-06T14:20:37.723 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.compression_opts.level: 32767 2024-09-06T14:20:37.723 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.compression_opts.strategy: 0 2024-09-06T14:20:37.723 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-06T14:20:37.723 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-06T14:20:37.723 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-06T14:20:37.723 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-06T14:20:37.723 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.compression_opts.enabled: false 2024-09-06T14:20:37.723 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-06T14:20:37.723 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-06T14:20:37.723 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-06T14:20:37.723 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-06T14:20:37.723 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.target_file_size_base: 67108864 2024-09-06T14:20:37.724 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-06T14:20:37.724 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-06T14:20:37.724 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-06T14:20:37.724 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-06T14:20:37.724 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-06T14:20:37.724 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-06T14:20:37.724 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-06T14:20:37.724 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-06T14:20:37.724 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-06T14:20:37.724 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-06T14:20:37.724 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-06T14:20:37.724 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-06T14:20:37.724 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-06T14:20:37.725 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-06T14:20:37.725 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.arena_block_size: 1048576 2024-09-06T14:20:37.725 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-06T14:20:37.725 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-06T14:20:37.725 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.disable_auto_compactions: 0 2024-09-06T14:20:37.725 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-06T14:20:37.725 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-06T14:20:37.725 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-06T14:20:37.725 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-06T14:20:37.725 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-06T14:20:37.725 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-06T14:20:37.726 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-06T14:20:37.726 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-06T14:20:37.726 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-06T14:20:37.726 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-06T14:20:37.726 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-06T14:20:37.726 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.inplace_update_support: 0 2024-09-06T14:20:37.726 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-06T14:20:37.726 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-06T14:20:37.726 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-06T14:20:37.726 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-06T14:20:37.726 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.bloom_locality: 0 2024-09-06T14:20:37.726 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.max_successive_merges: 0 2024-09-06T14:20:37.726 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-06T14:20:37.726 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.paranoid_file_checks: 0 2024-09-06T14:20:37.727 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.force_consistency_checks: 1 2024-09-06T14:20:37.727 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.report_bg_io_stats: 0 2024-09-06T14:20:37.727 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.ttl: 2592000 2024-09-06T14:20:37.727 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-06T14:20:37.727 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-06T14:20:37.727 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-06T14:20:37.727 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.enable_blob_files: false 2024-09-06T14:20:37.727 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.min_blob_size: 0 2024-09-06T14:20:37.727 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.blob_file_size: 268435456 2024-09-06T14:20:37.727 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-06T14:20:37.727 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-06T14:20:37.727 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-06T14:20:37.727 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-06T14:20:37.727 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-06T14:20:37.727 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.blob_file_starting_level: 0 2024-09-06T14:20:37.728 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-06T14:20:37.728 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-06T14:20:37.728 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-09-06T14:20:37.728 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: c70d464e-56f9-4111-a72f-4efcd68dbe3d 2024-09-06T14:20:37.728 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: EVENT_LOG_v1 {"time_micros": 1725632437432293, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-09-06T14:20:37.728 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-09-06T14:20:37.728 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: EVENT_LOG_v1 {"time_micros": 1725632437432667, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1773, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1725632437, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "c70d464e-56f9-4111-a72f-4efcd68dbe3d", "db_session_id": "FVN8X5F1A64HTU033PD1", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-09-06T14:20:37.728 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: EVENT_LOG_v1 {"time_micros": 1725632437432761, "job": 1, "event": "recovery_finished"} 2024-09-06T14:20:37.728 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-09-06T14:20:37.728 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-06T14:20:37.728 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55b71b786e00 2024-09-06T14:20:37.728 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: DB pointer 0x55b71b886000 2024-09-06T14:20:37.728 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-06T14:20:37.728 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-06T14:20:37.729 INFO:journalctl@ceph.mon.c.smithi142.stdout: ** DB Stats ** 2024-09-06T14:20:37.729 INFO:journalctl@ceph.mon.c.smithi142.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-06T14:20:37.729 INFO:journalctl@ceph.mon.c.smithi142.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-06T14:20:37.729 INFO:journalctl@ceph.mon.c.smithi142.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-06T14:20:37.729 INFO:journalctl@ceph.mon.c.smithi142.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-06T14:20:37.729 INFO:journalctl@ceph.mon.c.smithi142.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-06T14:20:37.729 INFO:journalctl@ceph.mon.c.smithi142.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-06T14:20:37.729 INFO:journalctl@ceph.mon.c.smithi142.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-06T14:20:37.729 INFO:journalctl@ceph.mon.c.smithi142.stdout: 2024-09-06T14:20:37.729 INFO:journalctl@ceph.mon.c.smithi142.stdout: ** Compaction Stats [default] ** 2024-09-06T14:20:37.729 INFO:journalctl@ceph.mon.c.smithi142.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-06T14:20:37.729 INFO:journalctl@ceph.mon.c.smithi142.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-06T14:20:37.729 INFO:journalctl@ceph.mon.c.smithi142.stdout: L0 1/0 1.73 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.7 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-06T14:20:37.730 INFO:journalctl@ceph.mon.c.smithi142.stdout: Sum 1/0 1.73 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.7 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-06T14:20:37.730 INFO:journalctl@ceph.mon.c.smithi142.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.7 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-06T14:20:37.730 INFO:journalctl@ceph.mon.c.smithi142.stdout: 2024-09-06T14:20:37.730 INFO:journalctl@ceph.mon.c.smithi142.stdout: ** Compaction Stats [default] ** 2024-09-06T14:20:37.730 INFO:journalctl@ceph.mon.c.smithi142.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-06T14:20:37.730 INFO:journalctl@ceph.mon.c.smithi142.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-06T14:20:37.730 INFO:journalctl@ceph.mon.c.smithi142.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.7 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-06T14:20:37.730 INFO:journalctl@ceph.mon.c.smithi142.stdout: 2024-09-06T14:20:37.730 INFO:journalctl@ceph.mon.c.smithi142.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-06T14:20:37.730 INFO:journalctl@ceph.mon.c.smithi142.stdout: 2024-09-06T14:20:37.730 INFO:journalctl@ceph.mon.c.smithi142.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-06T14:20:37.730 INFO:journalctl@ceph.mon.c.smithi142.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-06T14:20:37.730 INFO:journalctl@ceph.mon.c.smithi142.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-06T14:20:37.730 INFO:journalctl@ceph.mon.c.smithi142.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-06T14:20:37.731 INFO:journalctl@ceph.mon.c.smithi142.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-06T14:20:37.731 INFO:journalctl@ceph.mon.c.smithi142.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-06T14:20:37.731 INFO:journalctl@ceph.mon.c.smithi142.stdout: Cumulative compaction: 0.00 GB write, 0.68 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-06T14:20:37.731 INFO:journalctl@ceph.mon.c.smithi142.stdout: Interval compaction: 0.00 GB write, 0.68 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-06T14:20:37.731 INFO:journalctl@ceph.mon.c.smithi142.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-06T14:20:37.731 INFO:journalctl@ceph.mon.c.smithi142.stdout: Block cache BinnedLRUCache@0x55b71b756850#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-09-06T14:20:37.731 INFO:journalctl@ceph.mon.c.smithi142.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-09-06T14:20:37.731 INFO:journalctl@ceph.mon.c.smithi142.stdout: 2024-09-06T14:20:37.731 INFO:journalctl@ceph.mon.c.smithi142.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-06T14:20:37.731 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-09-06T14:20:37.731 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: using public_addrv [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-06T14:20:37.731 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: starting mon.c rank -1 at public addrs [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] at bind addrs [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid ce32462e-6c5a-11ef-bce4-c7b262605968 2024-09-06T14:20:37.731 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: mon.c@-1(???) e0 preinit fsid ce32462e-6c5a-11ef-bce4-c7b262605968 2024-09-06T14:20:37.731 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: mon.c@-1(synchronizing).mds e1 new map 2024-09-06T14:20:37.732 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: mon.c@-1(synchronizing).mds e1 print_map 2024-09-06T14:20:37.732 INFO:journalctl@ceph.mon.c.smithi142.stdout: e1 2024-09-06T14:20:37.732 INFO:journalctl@ceph.mon.c.smithi142.stdout: btime 2024-09-06T14:19:13:103085+0000 2024-09-06T14:20:37.732 INFO:journalctl@ceph.mon.c.smithi142.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-06T14:20:37.732 INFO:journalctl@ceph.mon.c.smithi142.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-06T14:20:37.732 INFO:journalctl@ceph.mon.c.smithi142.stdout: legacy client fscid: -1 2024-09-06T14:20:37.732 INFO:journalctl@ceph.mon.c.smithi142.stdout: 2024-09-06T14:20:37.732 INFO:journalctl@ceph.mon.c.smithi142.stdout: No filesystems configured 2024-09-06T14:20:37.732 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-09-06T14:20:37.732 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-09-06T14:20:37.732 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-09-06T14:20:37.732 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-09-06T14:20:37.732 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-09-06T14:20:37.732 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-09-06T14:20:37.733 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-06T14:20:37.733 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-06T14:20:37.733 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-06T14:20:37.733 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-06T14:20:37.733 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: Updating smithi162:/etc/ceph/ceph.conf 2024-09-06T14:20:37.733 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: Updating smithi162:/var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/config/ceph.conf 2024-09-06T14:20:37.733 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-06T14:20:37.733 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: Updating smithi162:/etc/ceph/ceph.client.admin.keyring 2024-09-06T14:20:37.733 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:37.733 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:37.733 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:37.733 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:37.733 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-06T14:20:37.733 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:20:37.734 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: Updating smithi162:/var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/config/ceph.client.admin.keyring 2024-09-06T14:20:37.734 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi142:172.21.15.142=a;smithi142:[v2:172.21.15.142:3301,v1:172.21.15.142:6790]=c;smithi162:172.21.15.162=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-06T14:20:37.734 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: Saving service mon spec with placement smithi142:172.21.15.142=a;smithi142:[v2:172.21.15.142:3301,v1:172.21.15.142:6790]=c;smithi162:172.21.15.162=b;count:3 2024-09-06T14:20:37.734 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: Deploying daemon mon.b on smithi162 2024-09-06T14:20:37.734 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-06T14:20:37.734 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: from='client.? 172.21.15.162:0/1632527596' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-06T14:20:37.734 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-06T14:20:37.734 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: from='client.? 172.21.15.162:0/1251782708' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-06T14:20:37.734 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-06T14:20:37.734 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: mon.a calling monitor election 2024-09-06T14:20:37.734 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-06T14:20:37.734 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-06T14:20:37.734 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-06T14:20:37.735 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-06T14:20:37.735 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: mon.b calling monitor election 2024-09-06T14:20:37.735 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-06T14:20:37.735 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-06T14:20:37.735 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-06T14:20:37.735 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-06T14:20:37.735 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-06T14:20:37.735 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: monmap epoch 2 2024-09-06T14:20:37.735 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: fsid ce32462e-6c5a-11ef-bce4-c7b262605968 2024-09-06T14:20:37.735 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: last_changed 2024-09-06T14:20:29.067692+0000 2024-09-06T14:20:37.735 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: created 2024-09-06T14:19:01.768092+0000 2024-09-06T14:20:37.735 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: min_mon_release 19 (squid) 2024-09-06T14:20:37.735 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: election_strategy: 1 2024-09-06T14:20:37.735 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: 0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-06T14:20:37.736 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: 1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-09-06T14:20:37.736 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: fsmap 2024-09-06T14:20:37.736 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: osdmap e4: 0 total, 0 up, 0 in 2024-09-06T14:20:37.736 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: mgrmap e13: y(active, since 35s) 2024-09-06T14:20:37.736 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: Health detail: HEALTH_WARN Module 'volumes' has failed dependency: No module named 'ceph.fs' 2024-09-06T14:20:37.736 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: [WRN] MGR_MODULE_DEPENDENCY: Module 'volumes' has failed dependency: No module named 'ceph.fs' 2024-09-06T14:20:37.736 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: Module 'volumes' has failed dependency: No module named 'ceph.fs' 2024-09-06T14:20:37.736 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:37.736 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:37.736 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:37.736 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-06T14:20:37.736 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:20:37.736 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: Deploying daemon mon.c on smithi142 2024-09-06T14:20:37.737 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: from='client.? 172.21.15.162:0/585470434' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-06T14:20:37.737 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-06T14:20:37.737 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-06T14:20:37.737 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 ceph-mon[37372]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-09-06T14:20:37.737 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 bash[37353]: 991d8632ae3f036d4a3b7ad111243e1bda9dd55176b080f99983007d32f94452 2024-09-06T14:20:37.737 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:37 smithi142 systemd[1]: Started Ceph mon.c for ce32462e-6c5a-11ef-bce4-c7b262605968. 2024-09-06T14:20:38.585 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-06T14:20:38.585 DEBUG:teuthology.orchestra.run.smithi162:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph mon dump -f json 2024-09-06T14:20:38.839 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.b/config 2024-09-06T14:20:42.796 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:42 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-06T14:20:42.797 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:42 smithi162 ceph-mon[33787]: mon.a calling monitor election 2024-09-06T14:20:42.797 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:42 smithi162 ceph-mon[33787]: mon.b calling monitor election 2024-09-06T14:20:42.797 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:42 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-06T14:20:42.797 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:42 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-06T14:20:42.798 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:42 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-06T14:20:42.798 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:42 smithi162 ceph-mon[33787]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-06T14:20:42.798 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:42 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-06T14:20:42.798 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:42 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-06T14:20:42.798 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:42 smithi162 ceph-mon[33787]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-06T14:20:42.799 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:42 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-06T14:20:42.799 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:42 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-06T14:20:42.799 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:42 smithi162 ceph-mon[33787]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-06T14:20:42.799 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:42 smithi162 ceph-mon[33787]: monmap epoch 3 2024-09-06T14:20:42.799 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:42 smithi162 ceph-mon[33787]: fsid ce32462e-6c5a-11ef-bce4-c7b262605968 2024-09-06T14:20:42.800 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:42 smithi162 ceph-mon[33787]: last_changed 2024-09-06T14:20:37.461046+0000 2024-09-06T14:20:42.800 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:42 smithi162 ceph-mon[33787]: created 2024-09-06T14:19:01.768092+0000 2024-09-06T14:20:42.800 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:42 smithi162 ceph-mon[33787]: min_mon_release 19 (squid) 2024-09-06T14:20:42.800 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:42 smithi162 ceph-mon[33787]: election_strategy: 1 2024-09-06T14:20:42.800 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:42 smithi162 ceph-mon[33787]: 0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-06T14:20:42.800 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:42 smithi162 ceph-mon[33787]: 1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-09-06T14:20:42.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:42 smithi162 ceph-mon[33787]: 2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-06T14:20:42.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:42 smithi162 ceph-mon[33787]: fsmap 2024-09-06T14:20:42.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:42 smithi162 ceph-mon[33787]: osdmap e4: 0 total, 0 up, 0 in 2024-09-06T14:20:42.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:42 smithi162 ceph-mon[33787]: mgrmap e13: y(active, since 43s) 2024-09-06T14:20:42.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:42 smithi162 ceph-mon[33787]: Health check failed: 1/3 mons down, quorum a,b (MON_DOWN) 2024-09-06T14:20:42.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:42 smithi162 ceph-mon[33787]: Health detail: HEALTH_WARN Module 'volumes' has failed dependency: No module named 'ceph.fs'; 1/3 mons down, quorum a,b 2024-09-06T14:20:42.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:42 smithi162 ceph-mon[33787]: [WRN] MGR_MODULE_DEPENDENCY: Module 'volumes' has failed dependency: No module named 'ceph.fs' 2024-09-06T14:20:42.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:42 smithi162 ceph-mon[33787]: Module 'volumes' has failed dependency: No module named 'ceph.fs' 2024-09-06T14:20:42.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:42 smithi162 ceph-mon[33787]: [WRN] MON_DOWN: 1/3 mons down, quorum a,b 2024-09-06T14:20:42.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:42 smithi162 ceph-mon[33787]: mon.c (rank 2) addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] is down (out of quorum) 2024-09-06T14:20:42.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:42 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:42.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:42 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:42.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:42 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:42.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:42 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:42.851 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:42.850+0000 7fdea340d640 1 -- 172.21.15.162:0/1181550523 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fde88003530 msgr2=0x7fde880059c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:20:42.851 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:42.850+0000 7fdea340d640 1 --2- 172.21.15.162:0/1181550523 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fde88003530 0x7fde880059c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fde9c102350 tx=0x7fde9802c3f0 comp rx=0 tx=0).stop 2024-09-06T14:20:42.851 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:42.850+0000 7fdea340d640 1 -- 172.21.15.162:0/1181550523 shutdown_connections 2024-09-06T14:20:42.851 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:42.850+0000 7fdea340d640 1 --2- 172.21.15.162:0/1181550523 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fde88003530 0x7fde880059c0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:20:42.851 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:42.850+0000 7fdea340d640 1 -- 172.21.15.162:0/1181550523 >> 172.21.15.162:0/1181550523 conn(0x7fde9c0fe070 msgr2=0x7fde9c100490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:20:42.852 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:42.850+0000 7fdea340d640 1 -- 172.21.15.162:0/1181550523 shutdown_connections 2024-09-06T14:20:42.852 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:42.850+0000 7fdea340d640 1 -- 172.21.15.162:0/1181550523 wait complete. 2024-09-06T14:20:42.852 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:42.851+0000 7fdea340d640 1 Processor -- start 2024-09-06T14:20:42.852 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:42.851+0000 7fdea340d640 1 -- start start 2024-09-06T14:20:42.852 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:42.851+0000 7fdea340d640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fde9c1977c0 0x7fde9c197ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:20:42.852 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:42.851+0000 7fdea340d640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fde9c1980e0 0x7fde9c19d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:20:42.852 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:42.851+0000 7fdea340d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fde9c19d6b0 0x7fde9c19db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:20:42.852 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:42.851+0000 7fdea340d640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fde9c109a10 con 0x7fde9c1980e0 2024-09-06T14:20:42.853 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:42.851+0000 7fdea340d640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fde9c109890 con 0x7fde9c19d6b0 2024-09-06T14:20:42.853 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:42.851+0000 7fdea340d640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7fde9c109b90 con 0x7fde9c1977c0 2024-09-06T14:20:42.853 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:42.851+0000 7fdea09a8640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fde9c1980e0 0x7fde9c19d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:20:42.853 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:42.851+0000 7fdea19aa640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fde9c19d6b0 0x7fde9c19db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:20:42.853 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:42.851+0000 7fdea09a8640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fde9c1980e0 0x7fde9c19d0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.162:58348/0 (socket says 172.21.15.162:58348) 2024-09-06T14:20:42.853 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:42.851+0000 7fdea09a8640 1 -- 172.21.15.162:0/139740630 learned_addr learned my addr 172.21.15.162:0/139740630 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-09-06T14:20:42.853 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:42.851+0000 7fdea11a9640 1 --2- 172.21.15.162:0/139740630 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fde9c1977c0 0x7fde9c197ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:20:42.853 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:42.851+0000 7fdea19aa640 1 -- 172.21.15.162:0/139740630 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fde9c19d6b0 msgr2=0x7fde9c19db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-09-06T14:20:42.853 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:42.851+0000 7fdea19aa640 1 -- 172.21.15.162:0/139740630 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fde9c19d6b0 msgr2=0x7fde9c19db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-06T14:20:42.853 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:42.851+0000 7fdea19aa640 1 --2- 172.21.15.162:0/139740630 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fde9c19d6b0 0x7fde9c19db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-06T14:20:42.854 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:42.851+0000 7fdea19aa640 1 --2- 172.21.15.162:0/139740630 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fde9c19d6b0 0x7fde9c19db20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-06T14:20:42.854 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:42.851+0000 7fdea11a9640 1 -- 172.21.15.162:0/139740630 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fde9c1977c0 msgr2=0x7fde9c197ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-09-06T14:20:42.854 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:42.851+0000 7fdea11a9640 1 -- 172.21.15.162:0/139740630 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fde9c1977c0 msgr2=0x7fde9c197ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-06T14:20:42.854 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:42.851+0000 7fdea11a9640 1 --2- 172.21.15.162:0/139740630 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fde9c1977c0 0x7fde9c197ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-06T14:20:42.854 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:42.851+0000 7fdea11a9640 1 --2- 172.21.15.162:0/139740630 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fde9c1977c0 0x7fde9c197ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-06T14:20:42.854 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:42.852+0000 7fdea09a8640 1 -- 172.21.15.162:0/139740630 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fde9c1977c0 msgr2=0x7fde9c197ba0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-06T14:20:42.854 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:42.852+0000 7fdea09a8640 1 --2- 172.21.15.162:0/139740630 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fde9c1977c0 0x7fde9c197ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:20:42.854 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:42.852+0000 7fdea09a8640 1 -- 172.21.15.162:0/139740630 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fde9c19d6b0 msgr2=0x7fde9c19db20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-06T14:20:42.854 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:42.852+0000 7fdea09a8640 1 --2- 172.21.15.162:0/139740630 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fde9c19d6b0 0x7fde9c19db20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:20:42.854 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:42.852+0000 7fdea09a8640 1 -- 172.21.15.162:0/139740630 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde9c1a4550 con 0x7fde9c1980e0 2024-09-06T14:20:42.854 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:42.852+0000 7fdea09a8640 1 --2- 172.21.15.162:0/139740630 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fde9c1980e0 0x7fde9c19d0a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fde8c00b520 tx=0x7fde8c00b9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:20:42.854 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:42.852+0000 7fde927fc640 1 -- 172.21.15.162:0/139740630 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde8c004230 con 0x7fde9c1980e0 2024-09-06T14:20:42.854 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:42.852+0000 7fde927fc640 1 -- 172.21.15.162:0/139740630 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7fde8c0043d0 con 0x7fde9c1980e0 2024-09-06T14:20:42.854 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:42.852+0000 7fdea340d640 1 -- 172.21.15.162:0/139740630 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde9c075140 con 0x7fde9c1980e0 2024-09-06T14:20:42.855 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:42.852+0000 7fde927fc640 1 -- 172.21.15.162:0/139740630 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde8c003bd0 con 0x7fde9c1980e0 2024-09-06T14:20:42.855 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:42.852+0000 7fdea340d640 1 -- 172.21.15.162:0/139740630 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde9c0756f0 con 0x7fde9c1980e0 2024-09-06T14:20:42.855 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:42.853+0000 7fde927fc640 1 -- 172.21.15.162:0/139740630 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 13) ==== 50132+0+0 (secure 0 0 0) 0x7fde8c00e070 con 0x7fde9c1980e0 2024-09-06T14:20:42.855 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:42.853+0000 7fde927fc640 1 --2- 172.21.15.162:0/139740630 >> v2:172.21.15.142:6800/853729507 conn(0x7fde8003df60 0x7fde80040420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:20:42.855 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:42.853+0000 7fde927fc640 1 -- 172.21.15.162:0/139740630 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 983+0+0 (secure 0 0 0) 0x7fde8c051700 con 0x7fde9c1980e0 2024-09-06T14:20:42.855 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:42.854+0000 7fdea11a9640 1 --2- 172.21.15.162:0/139740630 >> v2:172.21.15.142:6800/853729507 conn(0x7fde8003df60 0x7fde80040420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:20:42.855 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:42.854+0000 7fdea340d640 1 -- 172.21.15.162:0/139740630 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde9c077120 con 0x7fde9c1980e0 2024-09-06T14:20:42.855 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:42.854+0000 7fdea11a9640 1 --2- 172.21.15.162:0/139740630 >> v2:172.21.15.142:6800/853729507 conn(0x7fde8003df60 0x7fde80040420 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fde98005f10 tx=0x7fde98014930 comp rx=0 tx=0).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:20:42.856 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:42.859+0000 7fde927fc640 1 -- 172.21.15.162:0/139740630 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7fde8c014020 con 0x7fde9c1980e0 2024-09-06T14:20:42.958 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:42 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-06T14:20:42.958 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:42 smithi142 ceph-mon[29958]: mon.a calling monitor election 2024-09-06T14:20:42.958 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:42 smithi142 ceph-mon[29958]: mon.b calling monitor election 2024-09-06T14:20:42.958 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:42 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-06T14:20:42.958 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:42 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-06T14:20:42.958 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:42 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-06T14:20:42.958 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:42 smithi142 ceph-mon[29958]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-06T14:20:42.958 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:42 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-06T14:20:42.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:42 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-06T14:20:42.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:42 smithi142 ceph-mon[29958]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-06T14:20:42.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:42 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-06T14:20:42.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:42 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-06T14:20:42.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:42 smithi142 ceph-mon[29958]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-06T14:20:42.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:42 smithi142 ceph-mon[29958]: monmap epoch 3 2024-09-06T14:20:42.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:42 smithi142 ceph-mon[29958]: fsid ce32462e-6c5a-11ef-bce4-c7b262605968 2024-09-06T14:20:42.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:42 smithi142 ceph-mon[29958]: last_changed 2024-09-06T14:20:37.461046+0000 2024-09-06T14:20:42.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:42 smithi142 ceph-mon[29958]: created 2024-09-06T14:19:01.768092+0000 2024-09-06T14:20:42.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:42 smithi142 ceph-mon[29958]: min_mon_release 19 (squid) 2024-09-06T14:20:42.960 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:42 smithi142 ceph-mon[29958]: election_strategy: 1 2024-09-06T14:20:42.960 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:42 smithi142 ceph-mon[29958]: 0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-06T14:20:42.960 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:42 smithi142 ceph-mon[29958]: 1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-09-06T14:20:42.960 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:42 smithi142 ceph-mon[29958]: 2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-06T14:20:42.960 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:42 smithi142 ceph-mon[29958]: fsmap 2024-09-06T14:20:42.960 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:42 smithi142 ceph-mon[29958]: osdmap e4: 0 total, 0 up, 0 in 2024-09-06T14:20:42.961 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:42 smithi142 ceph-mon[29958]: mgrmap e13: y(active, since 43s) 2024-09-06T14:20:42.961 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:42 smithi142 ceph-mon[29958]: Health check failed: 1/3 mons down, quorum a,b (MON_DOWN) 2024-09-06T14:20:42.961 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:42 smithi142 ceph-mon[29958]: Health detail: HEALTH_WARN Module 'volumes' has failed dependency: No module named 'ceph.fs'; 1/3 mons down, quorum a,b 2024-09-06T14:20:42.961 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:42 smithi142 ceph-mon[29958]: [WRN] MGR_MODULE_DEPENDENCY: Module 'volumes' has failed dependency: No module named 'ceph.fs' 2024-09-06T14:20:42.961 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:42 smithi142 ceph-mon[29958]: Module 'volumes' has failed dependency: No module named 'ceph.fs' 2024-09-06T14:20:42.962 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:42 smithi142 ceph-mon[29958]: [WRN] MON_DOWN: 1/3 mons down, quorum a,b 2024-09-06T14:20:42.962 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:42 smithi142 ceph-mon[29958]: mon.c (rank 2) addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] is down (out of quorum) 2024-09-06T14:20:42.962 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:42 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:42.962 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:42 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:42.962 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:42 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:42.962 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:42 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:43.086 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:43.090+0000 7fdea340d640 1 -- 172.21.15.162:0/139740630 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7fde9c1988f0 con 0x7fde9c1980e0 2024-09-06T14:20:43.087 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:43.091+0000 7fde927fc640 1 -- 172.21.15.162:0/139740630 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1287 (secure 0 0 0) 0x7fde8c014200 con 0x7fde9c1980e0 2024-09-06T14:20:43.087 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-06T14:20:43.087 INFO:teuthology.orchestra.run.smithi162.stdout:{"epoch":3,"fsid":"ce32462e-6c5a-11ef-bce4-c7b262605968","modified":"2024-09-06T14:20:37.461046Z","created":"2024-09-06T14:19:01.768092Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:3300","nonce":0},{"type":"v1","addr":"172.21.15.162:6789","nonce":0}]},"addr":"172.21.15.162:6789/0","public_addr":"172.21.15.162:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-06T14:20:43.088 INFO:teuthology.orchestra.run.smithi162.stderr:dumped monmap epoch 3 2024-09-06T14:20:43.089 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:43.093+0000 7fdea340d640 1 -- 172.21.15.162:0/139740630 >> v2:172.21.15.142:6800/853729507 conn(0x7fde8003df60 msgr2=0x7fde80040420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:20:43.089 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:43.093+0000 7fdea340d640 1 --2- 172.21.15.162:0/139740630 >> v2:172.21.15.142:6800/853729507 conn(0x7fde8003df60 0x7fde80040420 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fde98005f10 tx=0x7fde98014930 comp rx=0 tx=0).stop 2024-09-06T14:20:43.089 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:43.093+0000 7fdea340d640 1 -- 172.21.15.162:0/139740630 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fde9c1980e0 msgr2=0x7fde9c19d0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:20:43.090 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:43.093+0000 7fdea340d640 1 --2- 172.21.15.162:0/139740630 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fde9c1980e0 0x7fde9c19d0a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fde8c00b520 tx=0x7fde8c00b9e0 comp rx=0 tx=0).stop 2024-09-06T14:20:43.090 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:43.093+0000 7fdea340d640 1 -- 172.21.15.162:0/139740630 shutdown_connections 2024-09-06T14:20:43.090 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:43.093+0000 7fdea340d640 1 --2- 172.21.15.162:0/139740630 >> v2:172.21.15.142:6800/853729507 conn(0x7fde8003df60 0x7fde80040420 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:20:43.090 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:43.093+0000 7fdea340d640 1 --2- 172.21.15.162:0/139740630 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fde9c19d6b0 0x7fde9c19db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:20:43.090 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:43.093+0000 7fdea340d640 1 --2- 172.21.15.162:0/139740630 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fde9c1980e0 0x7fde9c19d0a0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:20:43.090 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:43.093+0000 7fdea340d640 1 --2- 172.21.15.162:0/139740630 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fde9c1977c0 0x7fde9c197ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:20:43.090 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:43.093+0000 7fdea340d640 1 -- 172.21.15.162:0/139740630 >> 172.21.15.162:0/139740630 conn(0x7fde9c0fe070 msgr2=0x7fde9c1056c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:20:43.090 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:43.094+0000 7fdea340d640 1 -- 172.21.15.162:0/139740630 shutdown_connections 2024-09-06T14:20:43.090 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:43.094+0000 7fdea340d640 1 -- 172.21.15.162:0/139740630 wait complete. 2024-09-06T14:20:43.758 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:43 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-06T14:20:43.758 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:43 smithi162 ceph-mon[33787]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-06T14:20:43.758 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:43 smithi162 ceph-mon[33787]: from='client.? 172.21.15.162:0/139740630' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-06T14:20:43.758 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:43 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-06T14:20:43.786 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-09-06T14:20:43.787 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph config generate-minimal-conf 2024-09-06T14:20:43.810 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:43 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-06T14:20:43.810 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:43 smithi142 ceph-mon[29958]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-06T14:20:43.810 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:43 smithi142 ceph-mon[29958]: from='client.? 172.21.15.162:0/139740630' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-06T14:20:43.810 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:43 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-06T14:20:44.025 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:20:44.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:44.550+0000 7fb5bd853640 1 -- 172.21.15.142:0/844321434 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb5b806a5e0 msgr2=0x7fb5b806a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:20:44.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:44.550+0000 7fb5bd853640 1 --2- 172.21.15.142:0/844321434 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb5b806a5e0 0x7fb5b806a9c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fb5a8009880 tx=0x7fb5a802a5d0 comp rx=0 tx=0).stop 2024-09-06T14:20:44.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:44.550+0000 7fb5bd853640 1 -- 172.21.15.142:0/844321434 shutdown_connections 2024-09-06T14:20:44.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:44.550+0000 7fb5bd853640 1 --2- 172.21.15.142:0/844321434 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb5b806a5e0 0x7fb5b806a9c0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:20:44.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:44.550+0000 7fb5bd853640 1 -- 172.21.15.142:0/844321434 >> 172.21.15.142:0/844321434 conn(0x7fb5b8076ea0 msgr2=0x7fb5b80772b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:20:44.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:44.550+0000 7fb5bd853640 1 -- 172.21.15.142:0/844321434 shutdown_connections 2024-09-06T14:20:44.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:44.550+0000 7fb5bd853640 1 -- 172.21.15.142:0/844321434 wait complete. 2024-09-06T14:20:44.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:44.551+0000 7fb5bd853640 1 Processor -- start 2024-09-06T14:20:44.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:44.551+0000 7fb5bd853640 1 -- start start 2024-09-06T14:20:44.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:44.551+0000 7fb5bd853640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb5b806a5e0 0x7fb5b81975d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:20:44.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:44.551+0000 7fb5bd853640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb5b8197b10 0x7fb5b819ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:20:44.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:44.551+0000 7fb5bd853640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb5b819d3f0 0x7fb5b819d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:20:44.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:44.551+0000 7fb5bd853640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fb5b810bdf0 con 0x7fb5b806a5e0 2024-09-06T14:20:44.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:44.551+0000 7fb5bd853640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb5b810bc70 con 0x7fb5b8197b10 2024-09-06T14:20:44.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:44.551+0000 7fb5bd853640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7fb5b810bf70 con 0x7fb5b819d3f0 2024-09-06T14:20:44.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:44.551+0000 7fb5b6ffd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb5b806a5e0 0x7fb5b81975d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:20:44.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:44.551+0000 7fb5b6ffd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb5b806a5e0 0x7fb5b81975d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:49752/0 (socket says 172.21.15.142:49752) 2024-09-06T14:20:44.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:44.551+0000 7fb5b6ffd640 1 -- 172.21.15.142:0/3896666403 learned_addr learned my addr 172.21.15.142:0/3896666403 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:20:44.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:44.551+0000 7fb5b77fe640 1 --2- 172.21.15.142:0/3896666403 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb5b819d3f0 0x7fb5b819d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:20:44.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:44.551+0000 7fb5b67fc640 1 --2- 172.21.15.142:0/3896666403 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb5b8197b10 0x7fb5b819ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:20:44.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:44.552+0000 7fb5b6ffd640 1 -- 172.21.15.142:0/3896666403 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb5b819d3f0 msgr2=0x7fb5b819d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:20:44.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:44.552+0000 7fb5b6ffd640 1 --2- 172.21.15.142:0/3896666403 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb5b819d3f0 0x7fb5b819d8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:20:44.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:44.552+0000 7fb5b6ffd640 1 -- 172.21.15.142:0/3896666403 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb5b8197b10 msgr2=0x7fb5b819ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:20:44.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:44.552+0000 7fb5b6ffd640 1 --2- 172.21.15.142:0/3896666403 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb5b8197b10 0x7fb5b819ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:20:44.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:44.552+0000 7fb5b6ffd640 1 -- 172.21.15.142:0/3896666403 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5b81014f0 con 0x7fb5b806a5e0 2024-09-06T14:20:44.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:44.552+0000 7fb5b77fe640 1 --2- 172.21.15.142:0/3896666403 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb5b819d3f0 0x7fb5b819d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-06T14:20:44.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:44.552+0000 7fb5b6ffd640 1 --2- 172.21.15.142:0/3896666403 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb5b806a5e0 0x7fb5b81975d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fb5a8031440 tx=0x7fb5a8032bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:20:44.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:44.552+0000 7fb5bc851640 1 -- 172.21.15.142:0/3896666403 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5a803f030 con 0x7fb5b806a5e0 2024-09-06T14:20:44.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:44.552+0000 7fb5bc851640 1 -- 172.21.15.142:0/3896666403 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7fb5a80044c0 con 0x7fb5b806a5e0 2024-09-06T14:20:44.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:44.552+0000 7fb5bd853640 1 -- 172.21.15.142:0/3896666403 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5b8101780 con 0x7fb5b806a5e0 2024-09-06T14:20:44.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:44.552+0000 7fb5bc851640 1 -- 172.21.15.142:0/3896666403 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5a8035070 con 0x7fb5b806a5e0 2024-09-06T14:20:44.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:44.552+0000 7fb5bd853640 1 -- 172.21.15.142:0/3896666403 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5b8101cf0 con 0x7fb5b806a5e0 2024-09-06T14:20:44.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:44.553+0000 7fb5bc851640 1 -- 172.21.15.142:0/3896666403 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 13) ==== 50132+0+0 (secure 0 0 0) 0x7fb5a8042030 con 0x7fb5b806a5e0 2024-09-06T14:20:44.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:44.553+0000 7fb5bc851640 1 --2- 172.21.15.142:0/3896666403 >> v2:172.21.15.142:6800/853729507 conn(0x7fb5a003df60 0x7fb5a0040420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:20:44.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:44.553+0000 7fb5bc851640 1 -- 172.21.15.142:0/3896666403 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 983+0+0 (secure 0 0 0) 0x7fb5a8005000 con 0x7fb5b806a5e0 2024-09-06T14:20:44.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:44.554+0000 7fb5bd853640 1 -- 172.21.15.142:0/3896666403 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5b8103910 con 0x7fb5b806a5e0 2024-09-06T14:20:44.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:44.554+0000 7fb5b67fc640 1 --2- 172.21.15.142:0/3896666403 >> v2:172.21.15.142:6800/853729507 conn(0x7fb5a003df60 0x7fb5a0040420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:20:44.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:44.554+0000 7fb5b67fc640 1 --2- 172.21.15.142:0/3896666403 >> v2:172.21.15.142:6800/853729507 conn(0x7fb5a003df60 0x7fb5a0040420 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb5ac009a30 tx=0x7fb5ac009190 comp rx=0 tx=0).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:20:44.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:44.558+0000 7fb5bc851640 1 -- 172.21.15.142:0/3896666403 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7fb5a80052d0 con 0x7fb5b806a5e0 2024-09-06T14:20:44.707 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:44 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-06T14:20:44.707 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:44 smithi142 ceph-mon[37372]: mon.a calling monitor election 2024-09-06T14:20:44.707 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:44 smithi142 ceph-mon[37372]: mon.b calling monitor election 2024-09-06T14:20:44.708 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:44 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-06T14:20:44.708 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:44 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-06T14:20:44.708 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:44 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-06T14:20:44.708 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:44 smithi142 ceph-mon[37372]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-06T14:20:44.708 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:44 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-06T14:20:44.708 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:44 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-06T14:20:44.708 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:44 smithi142 ceph-mon[37372]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-06T14:20:44.708 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:44 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-06T14:20:44.708 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:44 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-06T14:20:44.708 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:44 smithi142 ceph-mon[37372]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-06T14:20:44.708 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:44 smithi142 ceph-mon[37372]: monmap epoch 3 2024-09-06T14:20:44.708 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:44 smithi142 ceph-mon[37372]: fsid ce32462e-6c5a-11ef-bce4-c7b262605968 2024-09-06T14:20:44.708 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:44 smithi142 ceph-mon[37372]: last_changed 2024-09-06T14:20:37.461046+0000 2024-09-06T14:20:44.708 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:44 smithi142 ceph-mon[37372]: created 2024-09-06T14:19:01.768092+0000 2024-09-06T14:20:44.709 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:44 smithi142 ceph-mon[37372]: min_mon_release 19 (squid) 2024-09-06T14:20:44.709 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:44 smithi142 ceph-mon[37372]: election_strategy: 1 2024-09-06T14:20:44.709 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:44 smithi142 ceph-mon[37372]: 0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-06T14:20:44.709 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:44 smithi142 ceph-mon[37372]: 1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-09-06T14:20:44.709 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:44 smithi142 ceph-mon[37372]: 2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-06T14:20:44.709 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:44 smithi142 ceph-mon[37372]: fsmap 2024-09-06T14:20:44.709 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:44 smithi142 ceph-mon[37372]: osdmap e4: 0 total, 0 up, 0 in 2024-09-06T14:20:44.709 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:44 smithi142 ceph-mon[37372]: mgrmap e13: y(active, since 43s) 2024-09-06T14:20:44.709 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:44 smithi142 ceph-mon[37372]: Health check failed: 1/3 mons down, quorum a,b (MON_DOWN) 2024-09-06T14:20:44.709 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:44 smithi142 ceph-mon[37372]: Health detail: HEALTH_WARN Module 'volumes' has failed dependency: No module named 'ceph.fs'; 1/3 mons down, quorum a,b 2024-09-06T14:20:44.709 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:44 smithi142 ceph-mon[37372]: [WRN] MGR_MODULE_DEPENDENCY: Module 'volumes' has failed dependency: No module named 'ceph.fs' 2024-09-06T14:20:44.709 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:44 smithi142 ceph-mon[37372]: Module 'volumes' has failed dependency: No module named 'ceph.fs' 2024-09-06T14:20:44.709 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:44 smithi142 ceph-mon[37372]: [WRN] MON_DOWN: 1/3 mons down, quorum a,b 2024-09-06T14:20:44.709 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:44 smithi142 ceph-mon[37372]: mon.c (rank 2) addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] is down (out of quorum) 2024-09-06T14:20:44.710 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:44 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:44.710 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:44 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:44.710 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:44 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:44.710 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:44 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:44.710 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:44 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-06T14:20:44.710 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:44 smithi142 ceph-mon[37372]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-06T14:20:44.710 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:44 smithi142 ceph-mon[37372]: from='client.? 172.21.15.162:0/139740630' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-06T14:20:44.710 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:44 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-06T14:20:44.729 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:44.734+0000 7fb5bd853640 1 -- 172.21.15.142:0/3896666403 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7fb5b8198730 con 0x7fb5b806a5e0 2024-09-06T14:20:44.729 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:44.734+0000 7fb5bc851640 1 -- 172.21.15.142:0/3896666403 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+277 (secure 0 0 0) 0x7fb5a8003a60 con 0x7fb5b806a5e0 2024-09-06T14:20:44.730 INFO:teuthology.orchestra.run.smithi142.stdout:# minimal ceph.conf for ce32462e-6c5a-11ef-bce4-c7b262605968 2024-09-06T14:20:44.730 INFO:teuthology.orchestra.run.smithi142.stdout:[global] 2024-09-06T14:20:44.730 INFO:teuthology.orchestra.run.smithi142.stdout: fsid = ce32462e-6c5a-11ef-bce4-c7b262605968 2024-09-06T14:20:44.730 INFO:teuthology.orchestra.run.smithi142.stdout: mon_host = [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-06T14:20:44.731 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:44.736+0000 7fb5bd853640 1 -- 172.21.15.142:0/3896666403 >> v2:172.21.15.142:6800/853729507 conn(0x7fb5a003df60 msgr2=0x7fb5a0040420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:20:44.732 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:44.736+0000 7fb5bd853640 1 --2- 172.21.15.142:0/3896666403 >> v2:172.21.15.142:6800/853729507 conn(0x7fb5a003df60 0x7fb5a0040420 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb5ac009a30 tx=0x7fb5ac009190 comp rx=0 tx=0).stop 2024-09-06T14:20:44.732 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:44.736+0000 7fb5bd853640 1 -- 172.21.15.142:0/3896666403 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb5b806a5e0 msgr2=0x7fb5b81975d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:20:44.732 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:44.736+0000 7fb5bd853640 1 --2- 172.21.15.142:0/3896666403 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb5b806a5e0 0x7fb5b81975d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fb5a8031440 tx=0x7fb5a8032bf0 comp rx=0 tx=0).stop 2024-09-06T14:20:44.732 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:44.736+0000 7fb5bd853640 1 -- 172.21.15.142:0/3896666403 shutdown_connections 2024-09-06T14:20:44.732 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:44.736+0000 7fb5bd853640 1 --2- 172.21.15.142:0/3896666403 >> v2:172.21.15.142:6800/853729507 conn(0x7fb5a003df60 0x7fb5a0040420 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:20:44.732 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:44.736+0000 7fb5bd853640 1 --2- 172.21.15.142:0/3896666403 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb5b819d3f0 0x7fb5b819d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:20:44.732 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:44.736+0000 7fb5bd853640 1 --2- 172.21.15.142:0/3896666403 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb5b8197b10 0x7fb5b819ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:20:44.732 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:44.736+0000 7fb5bd853640 1 --2- 172.21.15.142:0/3896666403 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb5b806a5e0 0x7fb5b81975d0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:20:44.732 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:44.736+0000 7fb5bd853640 1 -- 172.21.15.142:0/3896666403 >> 172.21.15.142:0/3896666403 conn(0x7fb5b8076ea0 msgr2=0x7fb5b81085f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:20:44.732 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:44.736+0000 7fb5bd853640 1 -- 172.21.15.142:0/3896666403 shutdown_connections 2024-09-06T14:20:44.732 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:44.736+0000 7fb5bd853640 1 -- 172.21.15.142:0/3896666403 wait complete. 2024-09-06T14:20:45.329 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:45 smithi142 ceph-mon[29958]: mon.c calling monitor election 2024-09-06T14:20:45.329 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:45 smithi142 ceph-mon[29958]: Updating smithi142:/etc/ceph/ceph.conf 2024-09-06T14:20:45.329 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:45 smithi142 ceph-mon[29958]: Updating smithi162:/etc/ceph/ceph.conf 2024-09-06T14:20:45.329 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:45 smithi142 ceph-mon[29958]: Updating smithi142:/var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/config/ceph.conf 2024-09-06T14:20:45.330 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:45 smithi142 ceph-mon[29958]: mon.c calling monitor election 2024-09-06T14:20:45.330 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:45 smithi142 ceph-mon[29958]: mon.b calling monitor election 2024-09-06T14:20:45.330 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:45 smithi142 ceph-mon[29958]: mon.a calling monitor election 2024-09-06T14:20:45.330 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:45 smithi142 ceph-mon[29958]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-06T14:20:45.330 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:45 smithi142 ceph-mon[29958]: Updating smithi162:/var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/config/ceph.conf 2024-09-06T14:20:45.331 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:45 smithi142 ceph-mon[29958]: monmap epoch 3 2024-09-06T14:20:45.331 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:45 smithi142 ceph-mon[29958]: fsid ce32462e-6c5a-11ef-bce4-c7b262605968 2024-09-06T14:20:45.331 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:45 smithi142 ceph-mon[29958]: last_changed 2024-09-06T14:20:37.461046+0000 2024-09-06T14:20:45.331 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:45 smithi142 ceph-mon[29958]: created 2024-09-06T14:19:01.768092+0000 2024-09-06T14:20:45.331 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:45 smithi142 ceph-mon[29958]: min_mon_release 19 (squid) 2024-09-06T14:20:45.331 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:45 smithi142 ceph-mon[29958]: election_strategy: 1 2024-09-06T14:20:45.332 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:45 smithi142 ceph-mon[29958]: 0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-06T14:20:45.332 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:45 smithi142 ceph-mon[29958]: 1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-09-06T14:20:45.332 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:45 smithi142 ceph-mon[29958]: 2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-06T14:20:45.332 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:45 smithi142 ceph-mon[29958]: fsmap 2024-09-06T14:20:45.332 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:45 smithi142 ceph-mon[29958]: osdmap e4: 0 total, 0 up, 0 in 2024-09-06T14:20:45.333 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:45 smithi142 ceph-mon[29958]: mgrmap e13: y(active, since 45s) 2024-09-06T14:20:45.333 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:45 smithi142 ceph-mon[29958]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2024-09-06T14:20:45.333 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:45 smithi142 ceph-mon[29958]: Health detail: HEALTH_WARN Module 'volumes' has failed dependency: No module named 'ceph.fs' 2024-09-06T14:20:45.333 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:45 smithi142 ceph-mon[29958]: [WRN] MGR_MODULE_DEPENDENCY: Module 'volumes' has failed dependency: No module named 'ceph.fs' 2024-09-06T14:20:45.333 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:45 smithi142 ceph-mon[29958]: Module 'volumes' has failed dependency: No module named 'ceph.fs' 2024-09-06T14:20:45.334 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:45 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:45.334 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:45 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:45.334 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:45 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:45.334 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:45 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:45.334 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:45 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:45.335 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:45 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/3896666403' entity='client.admin' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:20:45.335 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:45 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:45.335 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:45 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:45.335 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:45 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:45.335 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:45 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:45.336 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:45 smithi142 ceph-mon[29958]: Reconfiguring mon.a (unknown last config time)... 2024-09-06T14:20:45.336 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:45 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-06T14:20:45.336 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:45 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-06T14:20:45.336 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:45 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:20:45.336 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:45 smithi142 ceph-mon[29958]: Reconfiguring daemon mon.a on smithi142 2024-09-06T14:20:45.337 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:45 smithi142 ceph-mon[29958]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-06T14:20:45.337 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:45 smithi142 ceph-mon[37372]: mon.c calling monitor election 2024-09-06T14:20:45.337 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:45 smithi142 ceph-mon[37372]: Updating smithi142:/etc/ceph/ceph.conf 2024-09-06T14:20:45.338 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:45 smithi142 ceph-mon[37372]: Updating smithi162:/etc/ceph/ceph.conf 2024-09-06T14:20:45.338 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:45 smithi142 ceph-mon[37372]: Updating smithi142:/var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/config/ceph.conf 2024-09-06T14:20:45.338 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:45 smithi142 ceph-mon[37372]: mon.c calling monitor election 2024-09-06T14:20:45.338 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:45 smithi142 ceph-mon[37372]: mon.b calling monitor election 2024-09-06T14:20:45.338 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:45 smithi142 ceph-mon[37372]: mon.a calling monitor election 2024-09-06T14:20:45.338 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:45 smithi142 ceph-mon[37372]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-06T14:20:45.338 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:45 smithi142 ceph-mon[37372]: Updating smithi162:/var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/config/ceph.conf 2024-09-06T14:20:45.339 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:45 smithi142 ceph-mon[37372]: monmap epoch 3 2024-09-06T14:20:45.339 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:45 smithi142 ceph-mon[37372]: fsid ce32462e-6c5a-11ef-bce4-c7b262605968 2024-09-06T14:20:45.339 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:45 smithi142 ceph-mon[37372]: last_changed 2024-09-06T14:20:37.461046+0000 2024-09-06T14:20:45.339 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:45 smithi142 ceph-mon[37372]: created 2024-09-06T14:19:01.768092+0000 2024-09-06T14:20:45.339 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:45 smithi142 ceph-mon[37372]: min_mon_release 19 (squid) 2024-09-06T14:20:45.340 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:45 smithi142 ceph-mon[37372]: election_strategy: 1 2024-09-06T14:20:45.340 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:45 smithi142 ceph-mon[37372]: 0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-06T14:20:45.340 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:45 smithi142 ceph-mon[37372]: 1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-09-06T14:20:45.340 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:45 smithi142 ceph-mon[37372]: 2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-06T14:20:45.340 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:45 smithi142 ceph-mon[37372]: fsmap 2024-09-06T14:20:45.340 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:45 smithi142 ceph-mon[37372]: osdmap e4: 0 total, 0 up, 0 in 2024-09-06T14:20:45.340 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:45 smithi142 ceph-mon[37372]: mgrmap e13: y(active, since 45s) 2024-09-06T14:20:45.340 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:45 smithi142 ceph-mon[37372]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2024-09-06T14:20:45.341 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:45 smithi142 ceph-mon[37372]: Health detail: HEALTH_WARN Module 'volumes' has failed dependency: No module named 'ceph.fs' 2024-09-06T14:20:45.341 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:45 smithi142 ceph-mon[37372]: [WRN] MGR_MODULE_DEPENDENCY: Module 'volumes' has failed dependency: No module named 'ceph.fs' 2024-09-06T14:20:45.341 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:45 smithi142 ceph-mon[37372]: Module 'volumes' has failed dependency: No module named 'ceph.fs' 2024-09-06T14:20:45.341 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:45 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:45.341 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:45 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:45.341 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:45 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:45.342 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:45 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:45.342 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:45 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:45.342 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:45 smithi142 ceph-mon[37372]: from='client.? 172.21.15.142:0/3896666403' entity='client.admin' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:20:45.342 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:45 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:45.342 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:45 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:45.342 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:45 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:45.343 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:45 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:45.343 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:45 smithi142 ceph-mon[37372]: Reconfiguring mon.a (unknown last config time)... 2024-09-06T14:20:45.343 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:45 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-06T14:20:45.343 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:45 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-06T14:20:45.343 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:45 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:20:45.343 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:45 smithi142 ceph-mon[37372]: Reconfiguring daemon mon.a on smithi142 2024-09-06T14:20:45.343 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:45 smithi142 ceph-mon[37372]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-06T14:20:45.356 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-09-06T14:20:45.356 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-09-06T14:20:45.356 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd of=/etc/ceph/ceph.conf 2024-09-06T14:20:45.430 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-09-06T14:20:45.430 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-06T14:20:45.495 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-09-06T14:20:45.495 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/etc/ceph/ceph.conf 2024-09-06T14:20:45.520 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:45 smithi162 ceph-mon[33787]: mon.c calling monitor election 2024-09-06T14:20:45.520 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:45 smithi162 ceph-mon[33787]: Updating smithi142:/etc/ceph/ceph.conf 2024-09-06T14:20:45.520 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:45 smithi162 ceph-mon[33787]: Updating smithi162:/etc/ceph/ceph.conf 2024-09-06T14:20:45.520 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:45 smithi162 ceph-mon[33787]: Updating smithi142:/var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/config/ceph.conf 2024-09-06T14:20:45.520 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:45 smithi162 ceph-mon[33787]: mon.c calling monitor election 2024-09-06T14:20:45.520 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:45 smithi162 ceph-mon[33787]: mon.b calling monitor election 2024-09-06T14:20:45.520 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:45 smithi162 ceph-mon[33787]: mon.a calling monitor election 2024-09-06T14:20:45.520 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:45 smithi162 ceph-mon[33787]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-06T14:20:45.521 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:45 smithi162 ceph-mon[33787]: Updating smithi162:/var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/config/ceph.conf 2024-09-06T14:20:45.521 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:45 smithi162 ceph-mon[33787]: monmap epoch 3 2024-09-06T14:20:45.521 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:45 smithi162 ceph-mon[33787]: fsid ce32462e-6c5a-11ef-bce4-c7b262605968 2024-09-06T14:20:45.521 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:45 smithi162 ceph-mon[33787]: last_changed 2024-09-06T14:20:37.461046+0000 2024-09-06T14:20:45.521 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:45 smithi162 ceph-mon[33787]: created 2024-09-06T14:19:01.768092+0000 2024-09-06T14:20:45.521 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:45 smithi162 ceph-mon[33787]: min_mon_release 19 (squid) 2024-09-06T14:20:45.521 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:45 smithi162 ceph-mon[33787]: election_strategy: 1 2024-09-06T14:20:45.521 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:45 smithi162 ceph-mon[33787]: 0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-06T14:20:45.521 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:45 smithi162 ceph-mon[33787]: 1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-09-06T14:20:45.521 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:45 smithi162 ceph-mon[33787]: 2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-06T14:20:45.521 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:45 smithi162 ceph-mon[33787]: fsmap 2024-09-06T14:20:45.521 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:45 smithi162 ceph-mon[33787]: osdmap e4: 0 total, 0 up, 0 in 2024-09-06T14:20:45.521 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:45 smithi162 ceph-mon[33787]: mgrmap e13: y(active, since 45s) 2024-09-06T14:20:45.521 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:45 smithi162 ceph-mon[33787]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2024-09-06T14:20:45.522 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:45 smithi162 ceph-mon[33787]: Health detail: HEALTH_WARN Module 'volumes' has failed dependency: No module named 'ceph.fs' 2024-09-06T14:20:45.522 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:45 smithi162 ceph-mon[33787]: [WRN] MGR_MODULE_DEPENDENCY: Module 'volumes' has failed dependency: No module named 'ceph.fs' 2024-09-06T14:20:45.522 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:45 smithi162 ceph-mon[33787]: Module 'volumes' has failed dependency: No module named 'ceph.fs' 2024-09-06T14:20:45.522 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:45 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:45.522 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:45 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:45.522 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:45 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:45.522 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:45 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:45.522 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:45 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:45.522 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:45 smithi162 ceph-mon[33787]: from='client.? 172.21.15.142:0/3896666403' entity='client.admin' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:20:45.522 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:45 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:45.522 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:45 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:45.522 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:45 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:45.522 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:45 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:45.522 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:45 smithi162 ceph-mon[33787]: Reconfiguring mon.a (unknown last config time)... 2024-09-06T14:20:45.523 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:45 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-06T14:20:45.523 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:45 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-06T14:20:45.523 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:45 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:20:45.523 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:45 smithi162 ceph-mon[33787]: Reconfiguring daemon mon.a on smithi142 2024-09-06T14:20:45.523 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:45 smithi162 ceph-mon[33787]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-06T14:20:45.524 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-09-06T14:20:45.524 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-06T14:20:45.590 INFO:tasks.cephadm:Adding mgr.y on smithi142 2024-09-06T14:20:45.590 INFO:tasks.cephadm:Adding mgr.x on smithi162 2024-09-06T14:20:45.591 DEBUG:teuthology.orchestra.run.smithi162:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph orch apply mgr '2;smithi142=y;smithi162=x' 2024-09-06T14:20:45.870 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.b/config 2024-09-06T14:20:46.289 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:46 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-06T14:20:46.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:46.292+0000 7f21b224c640 1 -- 172.21.15.162:0/1789387550 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f21ac1011b0 msgr2=0x7f21ac1015b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:20:46.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:46.292+0000 7f21b224c640 1 --2- 172.21.15.162:0/1789387550 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f21ac1011b0 0x7f21ac1015b0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f219c009880 tx=0x7f219c02a5d0 comp rx=0 tx=0).stop 2024-09-06T14:20:46.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:46.292+0000 7f21b224c640 1 -- 172.21.15.162:0/1789387550 shutdown_connections 2024-09-06T14:20:46.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:46.292+0000 7f21b224c640 1 --2- 172.21.15.162:0/1789387550 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f21ac1011b0 0x7f21ac1015b0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:20:46.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:46.292+0000 7f21b224c640 1 -- 172.21.15.162:0/1789387550 >> 172.21.15.162:0/1789387550 conn(0x7f21ac0fca80 msgr2=0x7f21ac0feec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:20:46.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:46.292+0000 7f21b224c640 1 -- 172.21.15.162:0/1789387550 shutdown_connections 2024-09-06T14:20:46.307 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:46.292+0000 7f21b224c640 1 -- 172.21.15.162:0/1789387550 wait complete. 2024-09-06T14:20:46.307 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:46.293+0000 7f21b224c640 1 Processor -- start 2024-09-06T14:20:46.307 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:46.293+0000 7f21b224c640 1 -- start start 2024-09-06T14:20:46.307 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:46.293+0000 7f21b224c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f21ac1011b0 0x7f21ac0fe800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:20:46.307 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:46.293+0000 7f21b224c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f21ac0fd070 0x7f21ac0fd4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:20:46.307 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:46.293+0000 7f21b224c640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f21ac0fda30 0x7f21ac1a1260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:20:46.307 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:46.293+0000 7f21b224c640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f21ac106860 con 0x7f21ac0fd070 2024-09-06T14:20:46.308 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:46.293+0000 7f21b224c640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f21ac1066e0 con 0x7f21ac1011b0 2024-09-06T14:20:46.308 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:46.293+0000 7f21b224c640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7f21ac1069e0 con 0x7f21ac0fda30 2024-09-06T14:20:46.308 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:46.293+0000 7f21ab7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f21ac1011b0 0x7f21ac0fe800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:20:46.308 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:46.293+0000 7f21a2ffd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f21ac0fd070 0x7f21ac0fd4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:20:46.308 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:46.293+0000 7f21ab7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f21ac1011b0 0x7f21ac0fe800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:47696/0 (socket says 172.21.15.162:47696) 2024-09-06T14:20:46.308 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:46.293+0000 7f21ab7fe640 1 -- 172.21.15.162:0/1537381825 learned_addr learned my addr 172.21.15.162:0/1537381825 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-09-06T14:20:46.308 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:46.293+0000 7f21abfff640 1 --2- 172.21.15.162:0/1537381825 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f21ac0fda30 0x7f21ac1a1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:20:46.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:46.294+0000 7f21a2ffd640 1 -- 172.21.15.162:0/1537381825 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f21ac0fda30 msgr2=0x7f21ac1a1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:20:46.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:46.294+0000 7f21a2ffd640 1 --2- 172.21.15.162:0/1537381825 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f21ac0fda30 0x7f21ac1a1260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:20:46.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:46.294+0000 7f21a2ffd640 1 -- 172.21.15.162:0/1537381825 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f21ac1011b0 msgr2=0x7f21ac0fe800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:20:46.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:46.294+0000 7f21a2ffd640 1 --2- 172.21.15.162:0/1537381825 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f21ac1011b0 0x7f21ac0fe800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:20:46.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:46.294+0000 7f21a2ffd640 1 -- 172.21.15.162:0/1537381825 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21ac1a17d0 con 0x7f21ac0fd070 2024-09-06T14:20:46.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:46.294+0000 7f21ab7fe640 1 --2- 172.21.15.162:0/1537381825 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f21ac1011b0 0x7f21ac0fe800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-06T14:20:46.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:46.294+0000 7f21abfff640 1 --2- 172.21.15.162:0/1537381825 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f21ac0fda30 0x7f21ac1a1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-06T14:20:46.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:46.294+0000 7f21a2ffd640 1 --2- 172.21.15.162:0/1537381825 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f21ac0fd070 0x7f21ac0fd4f0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f219400b450 tx=0x7f219400b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:20:46.310 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:46.294+0000 7f21a97fa640 1 -- 172.21.15.162:0/1537381825 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2194004230 con 0x7f21ac0fd070 2024-09-06T14:20:46.310 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:46.294+0000 7f21a97fa640 1 -- 172.21.15.162:0/1537381825 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7f21940043d0 con 0x7f21ac0fd070 2024-09-06T14:20:46.310 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:46.294+0000 7f21b224c640 1 -- 172.21.15.162:0/1537381825 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21ac1a1ac0 con 0x7f21ac0fd070 2024-09-06T14:20:46.310 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:46.294+0000 7f21a97fa640 1 -- 172.21.15.162:0/1537381825 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2194003a80 con 0x7f21ac0fd070 2024-09-06T14:20:46.310 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:46.294+0000 7f21b224c640 1 -- 172.21.15.162:0/1537381825 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21ac1a2050 con 0x7f21ac0fd070 2024-09-06T14:20:46.310 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:46.295+0000 7f21a97fa640 1 -- 172.21.15.162:0/1537381825 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 13) ==== 50132+0+0 (secure 0 0 0) 0x7f2194003c60 con 0x7f21ac0fd070 2024-09-06T14:20:46.310 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:46.295+0000 7f21a97fa640 1 --2- 172.21.15.162:0/1537381825 >> v2:172.21.15.142:6800/853729507 conn(0x7f219803df60 0x7f2198040420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:20:46.310 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:46.295+0000 7f21b224c640 1 -- 172.21.15.162:0/1537381825 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21ac1015b0 con 0x7f21ac0fd070 2024-09-06T14:20:46.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:46.295+0000 7f21a97fa640 1 -- 172.21.15.162:0/1537381825 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 983+0+0 (secure 0 0 0) 0x7f21940517f0 con 0x7f21ac0fd070 2024-09-06T14:20:46.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:46.295+0000 7f21ab7fe640 1 --2- 172.21.15.162:0/1537381825 >> v2:172.21.15.142:6800/853729507 conn(0x7f219803df60 0x7f2198040420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:20:46.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:46.296+0000 7f21ab7fe640 1 --2- 172.21.15.162:0/1537381825 >> v2:172.21.15.142:6800/853729507 conn(0x7f219803df60 0x7f2198040420 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f219c0096c0 tx=0x7f219c02aad0 comp rx=0 tx=0).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:20:46.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:46.299+0000 7f21a97fa640 1 -- 172.21.15.162:0/1537381825 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7f219401f200 con 0x7f21ac0fd070 2024-09-06T14:20:46.458 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:46.461+0000 7f21b224c640 1 -- 172.21.15.162:0/1537381825 --> v2:172.21.15.142:6800/853729507 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi142=y;smithi162=x", "target": ["mon-mgr", ""]}) -- 0x7f21ac0632f0 con 0x7f219803df60 2024-09-06T14:20:46.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:46.467+0000 7f21a97fa640 1 -- 172.21.15.162:0/1537381825 <== mgr.14146 v2:172.21.15.142:6800/853729507 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f21ac0632f0 con 0x7f219803df60 2024-09-06T14:20:46.463 INFO:teuthology.orchestra.run.smithi162.stdout:Scheduled mgr update... 2024-09-06T14:20:46.465 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:46.469+0000 7f21b224c640 1 -- 172.21.15.162:0/1537381825 >> v2:172.21.15.142:6800/853729507 conn(0x7f219803df60 msgr2=0x7f2198040420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:20:46.465 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:46.469+0000 7f21b224c640 1 --2- 172.21.15.162:0/1537381825 >> v2:172.21.15.142:6800/853729507 conn(0x7f219803df60 0x7f2198040420 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f219c0096c0 tx=0x7f219c02aad0 comp rx=0 tx=0).stop 2024-09-06T14:20:46.465 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:46.469+0000 7f21b224c640 1 -- 172.21.15.162:0/1537381825 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f21ac0fd070 msgr2=0x7f21ac0fd4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:20:46.465 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:46.469+0000 7f21b224c640 1 --2- 172.21.15.162:0/1537381825 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f21ac0fd070 0x7f21ac0fd4f0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f219400b450 tx=0x7f219400b910 comp rx=0 tx=0).stop 2024-09-06T14:20:46.466 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:46.469+0000 7f21b224c640 1 -- 172.21.15.162:0/1537381825 shutdown_connections 2024-09-06T14:20:46.466 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:46.469+0000 7f21b224c640 1 --2- 172.21.15.162:0/1537381825 >> v2:172.21.15.142:6800/853729507 conn(0x7f219803df60 0x7f2198040420 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:20:46.466 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:46.469+0000 7f21b224c640 1 --2- 172.21.15.162:0/1537381825 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f21ac0fda30 0x7f21ac1a1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:20:46.466 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:46.469+0000 7f21b224c640 1 --2- 172.21.15.162:0/1537381825 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f21ac0fd070 0x7f21ac0fd4f0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:20:46.466 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:46.469+0000 7f21b224c640 1 --2- 172.21.15.162:0/1537381825 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f21ac1011b0 0x7f21ac0fe800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:20:46.466 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:46.469+0000 7f21b224c640 1 -- 172.21.15.162:0/1537381825 >> 172.21.15.162:0/1537381825 conn(0x7f21ac0fca80 msgr2=0x7f21ac06ac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:20:46.466 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:46.469+0000 7f21b224c640 1 -- 172.21.15.162:0/1537381825 shutdown_connections 2024-09-06T14:20:46.466 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:20:46.469+0000 7f21b224c640 1 -- 172.21.15.162:0/1537381825 wait complete. 2024-09-06T14:20:46.543 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:46 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-06T14:20:46.543 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:46 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-06T14:20:47.093 DEBUG:teuthology.orchestra.run.smithi162:mgr.x> sudo journalctl -f -n 0 -u ceph-ce32462e-6c5a-11ef-bce4-c7b262605968@mgr.x.service 2024-09-06T14:20:47.096 DEBUG:tasks.cephadm:set 0 configs 2024-09-06T14:20:47.096 INFO:tasks.cephadm:Deploying OSDs... 2024-09-06T14:20:47.096 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-09-06T14:20:47.096 DEBUG:teuthology.orchestra.run.smithi142:> dd if=/scratch_devs of=/dev/stdout 2024-09-06T14:20:47.112 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-06T14:20:47.113 DEBUG:teuthology.orchestra.run.smithi142:> stat /dev/vg_nvme/lv_1 2024-09-06T14:20:47.169 INFO:teuthology.orchestra.run.smithi142.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-06T14:20:47.169 INFO:teuthology.orchestra.run.smithi142.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-06T14:20:47.169 INFO:teuthology.orchestra.run.smithi142.stdout:Device: 5h/5d Inode: 976 Links: 1 2024-09-06T14:20:47.169 INFO:teuthology.orchestra.run.smithi142.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-06T14:20:47.169 INFO:teuthology.orchestra.run.smithi142.stdout:Context: system_u:object_r:device_t:s0 2024-09-06T14:20:47.169 INFO:teuthology.orchestra.run.smithi142.stdout:Access: 2024-09-06 14:20:36.890980132 +0000 2024-09-06T14:20:47.169 INFO:teuthology.orchestra.run.smithi142.stdout:Modify: 2024-09-06 14:15:31.465924478 +0000 2024-09-06T14:20:47.170 INFO:teuthology.orchestra.run.smithi142.stdout:Change: 2024-09-06 14:15:31.465924478 +0000 2024-09-06T14:20:47.170 INFO:teuthology.orchestra.run.smithi142.stdout: Birth: 2024-09-06 14:15:31.465924478 +0000 2024-09-06T14:20:47.170 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-06T14:20:47.236 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records in 2024-09-06T14:20:47.237 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records out 2024-09-06T14:20:47.237 INFO:teuthology.orchestra.run.smithi142.stderr:512 bytes copied, 0.000215999 s, 2.4 MB/s 2024-09-06T14:20:47.238 DEBUG:teuthology.orchestra.run.smithi142:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-06T14:20:47.295 DEBUG:teuthology.orchestra.run.smithi142:> stat /dev/vg_nvme/lv_2 2024-09-06T14:20:47.350 INFO:teuthology.orchestra.run.smithi142.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-06T14:20:47.350 INFO:teuthology.orchestra.run.smithi142.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-06T14:20:47.351 INFO:teuthology.orchestra.run.smithi142.stdout:Device: 5h/5d Inode: 971 Links: 1 2024-09-06T14:20:47.351 INFO:teuthology.orchestra.run.smithi142.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-06T14:20:47.351 INFO:teuthology.orchestra.run.smithi142.stdout:Context: system_u:object_r:device_t:s0 2024-09-06T14:20:47.351 INFO:teuthology.orchestra.run.smithi142.stdout:Access: 2024-09-06 14:20:36.890980132 +0000 2024-09-06T14:20:47.351 INFO:teuthology.orchestra.run.smithi142.stdout:Modify: 2024-09-06 14:15:31.464924498 +0000 2024-09-06T14:20:47.351 INFO:teuthology.orchestra.run.smithi142.stdout:Change: 2024-09-06 14:15:31.464924498 +0000 2024-09-06T14:20:47.351 INFO:teuthology.orchestra.run.smithi142.stdout: Birth: 2024-09-06 14:15:31.464924498 +0000 2024-09-06T14:20:47.351 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-06T14:20:47.414 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records in 2024-09-06T14:20:47.414 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records out 2024-09-06T14:20:47.414 INFO:teuthology.orchestra.run.smithi142.stderr:512 bytes copied, 0.000251152 s, 2.0 MB/s 2024-09-06T14:20:47.416 DEBUG:teuthology.orchestra.run.smithi142:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-06T14:20:47.473 DEBUG:teuthology.orchestra.run.smithi142:> stat /dev/vg_nvme/lv_3 2024-09-06T14:20:47.528 INFO:teuthology.orchestra.run.smithi142.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-06T14:20:47.528 INFO:teuthology.orchestra.run.smithi142.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-06T14:20:47.529 INFO:teuthology.orchestra.run.smithi142.stdout:Device: 5h/5d Inode: 979 Links: 1 2024-09-06T14:20:47.529 INFO:teuthology.orchestra.run.smithi142.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-06T14:20:47.529 INFO:teuthology.orchestra.run.smithi142.stdout:Context: system_u:object_r:device_t:s0 2024-09-06T14:20:47.529 INFO:teuthology.orchestra.run.smithi142.stdout:Access: 2024-09-06 14:20:36.890980132 +0000 2024-09-06T14:20:47.529 INFO:teuthology.orchestra.run.smithi142.stdout:Modify: 2024-09-06 14:15:31.465924478 +0000 2024-09-06T14:20:47.529 INFO:teuthology.orchestra.run.smithi142.stdout:Change: 2024-09-06 14:15:31.465924478 +0000 2024-09-06T14:20:47.529 INFO:teuthology.orchestra.run.smithi142.stdout: Birth: 2024-09-06 14:15:31.465924478 +0000 2024-09-06T14:20:47.529 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-06T14:20:47.592 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:47 smithi142 ceph-mon[37372]: from='client.14208 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi142=y;smithi162=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-06T14:20:47.592 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:47 smithi142 ceph-mon[37372]: Saving service mgr spec with placement smithi142=y;smithi162=x;count:2 2024-09-06T14:20:47.592 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:47 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:47.592 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:47 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:47.593 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:47 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:47.593 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:47 smithi142 ceph-mon[37372]: Reconfiguring mgr.y (unknown last config time)... 2024-09-06T14:20:47.593 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:47 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-06T14:20:47.593 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:47 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-06T14:20:47.593 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:47 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:20:47.593 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:47 smithi142 ceph-mon[37372]: Reconfiguring daemon mgr.y on smithi142 2024-09-06T14:20:47.593 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:47 smithi142 ceph-mon[37372]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-06T14:20:47.594 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:47 smithi142 ceph-mon[29958]: from='client.14208 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi142=y;smithi162=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-06T14:20:47.594 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:47 smithi142 ceph-mon[29958]: Saving service mgr spec with placement smithi142=y;smithi162=x;count:2 2024-09-06T14:20:47.594 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:47 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:47.594 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:47 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:47.594 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:47 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:47.594 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:47 smithi142 ceph-mon[29958]: Reconfiguring mgr.y (unknown last config time)... 2024-09-06T14:20:47.594 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:47 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-06T14:20:47.595 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:47 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-06T14:20:47.595 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:47 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:20:47.595 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:47 smithi142 ceph-mon[29958]: Reconfiguring daemon mgr.y on smithi142 2024-09-06T14:20:47.595 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:47 smithi142 ceph-mon[29958]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-06T14:20:47.596 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records in 2024-09-06T14:20:47.597 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records out 2024-09-06T14:20:47.597 INFO:teuthology.orchestra.run.smithi142.stderr:512 bytes copied, 0.000293211 s, 1.7 MB/s 2024-09-06T14:20:47.597 DEBUG:teuthology.orchestra.run.smithi142:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-06T14:20:47.654 DEBUG:teuthology.orchestra.run.smithi142:> stat /dev/vg_nvme/lv_4 2024-09-06T14:20:47.710 INFO:teuthology.orchestra.run.smithi142.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-06T14:20:47.710 INFO:teuthology.orchestra.run.smithi142.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-06T14:20:47.710 INFO:teuthology.orchestra.run.smithi142.stdout:Device: 5h/5d Inode: 966 Links: 1 2024-09-06T14:20:47.710 INFO:teuthology.orchestra.run.smithi142.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-06T14:20:47.710 INFO:teuthology.orchestra.run.smithi142.stdout:Context: system_u:object_r:device_t:s0 2024-09-06T14:20:47.711 INFO:teuthology.orchestra.run.smithi142.stdout:Access: 2024-09-06 14:20:36.890980132 +0000 2024-09-06T14:20:47.712 INFO:teuthology.orchestra.run.smithi142.stdout:Modify: 2024-09-06 14:15:31.464924498 +0000 2024-09-06T14:20:47.712 INFO:teuthology.orchestra.run.smithi142.stdout:Change: 2024-09-06 14:15:31.464924498 +0000 2024-09-06T14:20:47.712 INFO:teuthology.orchestra.run.smithi142.stdout: Birth: 2024-09-06 14:15:31.464924498 +0000 2024-09-06T14:20:47.712 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-06T14:20:47.774 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records in 2024-09-06T14:20:47.774 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records out 2024-09-06T14:20:47.774 INFO:teuthology.orchestra.run.smithi142.stderr:512 bytes copied, 0.000214478 s, 2.4 MB/s 2024-09-06T14:20:47.775 DEBUG:teuthology.orchestra.run.smithi142:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-06T14:20:47.795 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:47 smithi162 ceph-mon[33787]: from='client.14208 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi142=y;smithi162=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-06T14:20:47.796 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:47 smithi162 ceph-mon[33787]: Saving service mgr spec with placement smithi142=y;smithi162=x;count:2 2024-09-06T14:20:47.796 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:47 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:47.796 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:47 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:47.796 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:47 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:47.796 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:47 smithi162 ceph-mon[33787]: Reconfiguring mgr.y (unknown last config time)... 2024-09-06T14:20:47.796 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:47 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-06T14:20:47.796 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:47 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-06T14:20:47.796 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:47 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:20:47.796 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:47 smithi162 ceph-mon[33787]: Reconfiguring daemon mgr.y on smithi142 2024-09-06T14:20:47.796 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:47 smithi162 ceph-mon[33787]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-06T14:20:47.832 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-09-06T14:20:47.832 DEBUG:teuthology.orchestra.run.smithi162:> dd if=/scratch_devs of=/dev/stdout 2024-09-06T14:20:47.850 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-06T14:20:47.850 DEBUG:teuthology.orchestra.run.smithi162:> stat /dev/vg_nvme/lv_1 2024-09-06T14:20:47.906 INFO:teuthology.orchestra.run.smithi162.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-06T14:20:47.907 INFO:teuthology.orchestra.run.smithi162.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-06T14:20:47.907 INFO:teuthology.orchestra.run.smithi162.stdout:Device: 5h/5d Inode: 971 Links: 1 2024-09-06T14:20:47.907 INFO:teuthology.orchestra.run.smithi162.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-06T14:20:47.907 INFO:teuthology.orchestra.run.smithi162.stdout:Context: system_u:object_r:device_t:s0 2024-09-06T14:20:47.907 INFO:teuthology.orchestra.run.smithi162.stdout:Access: 2024-09-06 14:20:27.719837392 +0000 2024-09-06T14:20:47.907 INFO:teuthology.orchestra.run.smithi162.stdout:Modify: 2024-09-06 14:15:31.331476002 +0000 2024-09-06T14:20:47.907 INFO:teuthology.orchestra.run.smithi162.stdout:Change: 2024-09-06 14:15:31.331476002 +0000 2024-09-06T14:20:47.907 INFO:teuthology.orchestra.run.smithi162.stdout: Birth: 2024-09-06 14:15:31.331476002 +0000 2024-09-06T14:20:47.907 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-06T14:20:47.972 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records in 2024-09-06T14:20:47.972 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records out 2024-09-06T14:20:47.972 INFO:teuthology.orchestra.run.smithi162.stderr:512 bytes copied, 0.000222241 s, 2.3 MB/s 2024-09-06T14:20:47.973 DEBUG:teuthology.orchestra.run.smithi162:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-06T14:20:48.031 DEBUG:teuthology.orchestra.run.smithi162:> stat /dev/vg_nvme/lv_2 2024-09-06T14:20:48.088 INFO:teuthology.orchestra.run.smithi162.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-06T14:20:48.088 INFO:teuthology.orchestra.run.smithi162.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-06T14:20:48.088 INFO:teuthology.orchestra.run.smithi162.stdout:Device: 5h/5d Inode: 964 Links: 1 2024-09-06T14:20:48.088 INFO:teuthology.orchestra.run.smithi162.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-06T14:20:48.088 INFO:teuthology.orchestra.run.smithi162.stdout:Context: system_u:object_r:device_t:s0 2024-09-06T14:20:48.089 INFO:teuthology.orchestra.run.smithi162.stdout:Access: 2024-09-06 14:20:27.720837373 +0000 2024-09-06T14:20:48.089 INFO:teuthology.orchestra.run.smithi162.stdout:Modify: 2024-09-06 14:15:31.331476002 +0000 2024-09-06T14:20:48.089 INFO:teuthology.orchestra.run.smithi162.stdout:Change: 2024-09-06 14:15:31.331476002 +0000 2024-09-06T14:20:48.089 INFO:teuthology.orchestra.run.smithi162.stdout: Birth: 2024-09-06 14:15:31.331476002 +0000 2024-09-06T14:20:48.089 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-06T14:20:48.153 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records in 2024-09-06T14:20:48.153 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records out 2024-09-06T14:20:48.153 INFO:teuthology.orchestra.run.smithi162.stderr:512 bytes copied, 0.000201573 s, 2.5 MB/s 2024-09-06T14:20:48.154 DEBUG:teuthology.orchestra.run.smithi162:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-06T14:20:48.212 DEBUG:teuthology.orchestra.run.smithi162:> stat /dev/vg_nvme/lv_3 2024-09-06T14:20:48.267 INFO:teuthology.orchestra.run.smithi162.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-06T14:20:48.268 INFO:teuthology.orchestra.run.smithi162.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-06T14:20:48.268 INFO:teuthology.orchestra.run.smithi162.stdout:Device: 5h/5d Inode: 975 Links: 1 2024-09-06T14:20:48.268 INFO:teuthology.orchestra.run.smithi162.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-06T14:20:48.268 INFO:teuthology.orchestra.run.smithi162.stdout:Context: system_u:object_r:device_t:s0 2024-09-06T14:20:48.268 INFO:teuthology.orchestra.run.smithi162.stdout:Access: 2024-09-06 14:20:27.720837373 +0000 2024-09-06T14:20:48.268 INFO:teuthology.orchestra.run.smithi162.stdout:Modify: 2024-09-06 14:15:31.332475983 +0000 2024-09-06T14:20:48.268 INFO:teuthology.orchestra.run.smithi162.stdout:Change: 2024-09-06 14:15:31.332475983 +0000 2024-09-06T14:20:48.268 INFO:teuthology.orchestra.run.smithi162.stdout: Birth: 2024-09-06 14:15:31.332475983 +0000 2024-09-06T14:20:48.268 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-06T14:20:48.331 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records in 2024-09-06T14:20:48.331 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records out 2024-09-06T14:20:48.331 INFO:teuthology.orchestra.run.smithi162.stderr:512 bytes copied, 0.000198328 s, 2.6 MB/s 2024-09-06T14:20:48.332 DEBUG:teuthology.orchestra.run.smithi162:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-06T14:20:48.389 DEBUG:teuthology.orchestra.run.smithi162:> stat /dev/vg_nvme/lv_4 2024-09-06T14:20:48.445 INFO:teuthology.orchestra.run.smithi162.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-06T14:20:48.446 INFO:teuthology.orchestra.run.smithi162.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-06T14:20:48.446 INFO:teuthology.orchestra.run.smithi162.stdout:Device: 5h/5d Inode: 981 Links: 1 2024-09-06T14:20:48.446 INFO:teuthology.orchestra.run.smithi162.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-06T14:20:48.446 INFO:teuthology.orchestra.run.smithi162.stdout:Context: system_u:object_r:device_t:s0 2024-09-06T14:20:48.447 INFO:teuthology.orchestra.run.smithi162.stdout:Access: 2024-09-06 14:20:27.720837373 +0000 2024-09-06T14:20:48.447 INFO:teuthology.orchestra.run.smithi162.stdout:Modify: 2024-09-06 14:15:31.333475963 +0000 2024-09-06T14:20:48.447 INFO:teuthology.orchestra.run.smithi162.stdout:Change: 2024-09-06 14:15:31.333475963 +0000 2024-09-06T14:20:48.447 INFO:teuthology.orchestra.run.smithi162.stdout: Birth: 2024-09-06 14:15:31.333475963 +0000 2024-09-06T14:20:48.447 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-06T14:20:48.509 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records in 2024-09-06T14:20:48.509 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records out 2024-09-06T14:20:48.509 INFO:teuthology.orchestra.run.smithi162.stderr:512 bytes copied, 0.000194385 s, 2.6 MB/s 2024-09-06T14:20:48.510 DEBUG:teuthology.orchestra.run.smithi162:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-06T14:20:48.567 INFO:tasks.cephadm:Deploying osd.0 on smithi142 with /dev/vg_nvme/lv_4... 2024-09-06T14:20:48.567 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- lvm zap /dev/vg_nvme/lv_4 2024-09-06T14:20:49.431 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:49 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:49.431 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:49 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:49.431 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:49 smithi142 ceph-mon[29958]: Reconfiguring mon.c (monmap changed)... 2024-09-06T14:20:49.431 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:49 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-06T14:20:49.431 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:49 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-06T14:20:49.431 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:49 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:20:49.431 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:49 smithi142 ceph-mon[29958]: Reconfiguring daemon mon.c on smithi142 2024-09-06T14:20:49.431 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:49 smithi142 ceph-mon[29958]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-06T14:20:49.432 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:49 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:49.432 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:49 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:49.432 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:49 smithi142 ceph-mon[37372]: Reconfiguring mon.c (monmap changed)... 2024-09-06T14:20:49.432 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:49 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-06T14:20:49.432 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:49 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-06T14:20:49.432 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:49 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:20:49.432 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:49 smithi142 ceph-mon[37372]: Reconfiguring daemon mon.c on smithi142 2024-09-06T14:20:49.432 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:49 smithi142 ceph-mon[37372]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-06T14:20:49.545 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:49 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:49.546 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:49 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:49.546 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:49 smithi162 ceph-mon[33787]: Reconfiguring mon.c (monmap changed)... 2024-09-06T14:20:49.546 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:49 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-06T14:20:49.546 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:49 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-06T14:20:49.546 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:49 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:20:49.546 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:49 smithi162 ceph-mon[33787]: Reconfiguring daemon mon.c on smithi142 2024-09-06T14:20:49.546 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:49 smithi162 ceph-mon[33787]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-06T14:20:49.551 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:20:50.789 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:50 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:50.789 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:50 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:50.789 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:50 smithi162 ceph-mon[33787]: Reconfiguring mon.b (monmap changed)... 2024-09-06T14:20:50.790 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:50 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-06T14:20:50.790 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:50 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-06T14:20:50.790 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:50 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:20:50.790 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:50 smithi162 ceph-mon[33787]: Reconfiguring daemon mon.b on smithi162 2024-09-06T14:20:50.797 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:50 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:50.797 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:50 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:50.797 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:50 smithi142 ceph-mon[29958]: Reconfiguring mon.b (monmap changed)... 2024-09-06T14:20:50.797 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:50 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-06T14:20:50.798 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:50 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-06T14:20:50.798 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:50 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:20:50.798 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:50 smithi142 ceph-mon[29958]: Reconfiguring daemon mon.b on smithi162 2024-09-06T14:20:50.798 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:50 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:50.798 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:50 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:50.798 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:50 smithi142 ceph-mon[37372]: Reconfiguring mon.b (monmap changed)... 2024-09-06T14:20:50.798 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:50 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-06T14:20:50.798 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:50 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-06T14:20:50.798 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:50 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:20:50.798 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:50 smithi142 ceph-mon[37372]: Reconfiguring daemon mon.b on smithi162 2024-09-06T14:20:52.061 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:51 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:52.061 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:51 smithi142 ceph-mon[37372]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-06T14:20:52.061 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:51 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:52.061 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:51 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-06T14:20:52.061 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:51 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:20:52.061 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:51 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-06T14:20:52.061 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:51 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:52.061 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:51 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-06T14:20:52.062 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:51 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-06T14:20:52.062 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:51 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-06T14:20:52.062 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:51 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:20:52.062 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:51 smithi142 ceph-mon[37372]: Deploying daemon mgr.x on smithi162 2024-09-06T14:20:52.062 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:51 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:52.062 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:51 smithi142 ceph-mon[29958]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-06T14:20:52.062 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:51 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:52.062 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:51 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-06T14:20:52.062 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:51 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:20:52.062 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:51 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-06T14:20:52.063 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:51 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:52.063 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:51 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-06T14:20:52.063 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:51 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-06T14:20:52.063 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:51 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-06T14:20:52.063 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:51 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:20:52.063 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:51 smithi142 ceph-mon[29958]: Deploying daemon mgr.x on smithi162 2024-09-06T14:20:52.065 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:20:52.091 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph orch daemon add osd smithi142:vg_nvme/lv_4 2024-09-06T14:20:52.135 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:51 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:52.135 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:51 smithi162 ceph-mon[33787]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-06T14:20:52.136 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:51 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:52.136 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:51 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-06T14:20:52.136 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:51 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:20:52.136 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:51 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-06T14:20:52.136 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:51 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:52.136 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:51 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-06T14:20:52.136 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:51 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-06T14:20:52.136 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:51 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-06T14:20:52.136 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:51 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:20:52.136 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:51 smithi162 ceph-mon[33787]: Deploying daemon mgr.x on smithi162 2024-09-06T14:20:52.334 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:20:52.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:52.777+0000 7f6bd4b9e640 1 -- 172.21.15.142:0/3241617715 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6bd0105740 msgr2=0x7f6bd0107bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:20:52.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:52.777+0000 7f6bd4b9e640 1 --2- 172.21.15.142:0/3241617715 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6bd0105740 0x7f6bd0107bb0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f6bb8009900 tx=0x7f6bb802a410 comp rx=0 tx=0).stop 2024-09-06T14:20:52.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:52.777+0000 7f6bd4b9e640 1 -- 172.21.15.142:0/3241617715 shutdown_connections 2024-09-06T14:20:52.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:52.777+0000 7f6bd4b9e640 1 --2- 172.21.15.142:0/3241617715 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6bd01082f0 0x7f6bd010a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:20:52.782 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:52.777+0000 7f6bd4b9e640 1 --2- 172.21.15.142:0/3241617715 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6bd0105740 0x7f6bd0107bb0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:20:52.782 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:52.777+0000 7f6bd4b9e640 1 --2- 172.21.15.142:0/3241617715 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6bd006b410 0x7f6bd01051b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:20:52.782 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:52.777+0000 7f6bd4b9e640 1 -- 172.21.15.142:0/3241617715 >> 172.21.15.142:0/3241617715 conn(0x7f6bd00fcd80 msgr2=0x7f6bd00ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:20:52.782 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:52.777+0000 7f6bd4b9e640 1 -- 172.21.15.142:0/3241617715 shutdown_connections 2024-09-06T14:20:52.782 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:52.777+0000 7f6bd4b9e640 1 -- 172.21.15.142:0/3241617715 wait complete. 2024-09-06T14:20:52.782 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:52.777+0000 7f6bd4b9e640 1 Processor -- start 2024-09-06T14:20:52.782 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:52.778+0000 7f6bd4b9e640 1 -- start start 2024-09-06T14:20:52.782 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:52.778+0000 7f6bd4b9e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6bd006b410 0x7f6bd0198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:20:52.782 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:52.778+0000 7f6bd4b9e640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6bd0105740 0x7f6bd0198690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:20:52.783 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:52.778+0000 7f6bd4b9e640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6bd01082f0 0x7f6bd01992d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:20:52.783 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:52.778+0000 7f6bd4b9e640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f6bd010d950 con 0x7f6bd0105740 2024-09-06T14:20:52.783 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:52.778+0000 7f6bd4b9e640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6bd010d7d0 con 0x7f6bd006b410 2024-09-06T14:20:52.783 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:52.778+0000 7f6bd4b9e640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7f6bd010dad0 con 0x7f6bd01082f0 2024-09-06T14:20:52.783 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:52.778+0000 7f6bc6ffd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6bd0105740 0x7f6bd0198690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:20:52.783 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:52.778+0000 7f6bc6ffd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6bd0105740 0x7f6bd0198690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:43556/0 (socket says 172.21.15.142:43556) 2024-09-06T14:20:52.783 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:52.778+0000 7f6bc6ffd640 1 -- 172.21.15.142:0/3525357616 learned_addr learned my addr 172.21.15.142:0/3525357616 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:20:52.783 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:52.778+0000 7f6bcffff640 1 --2- 172.21.15.142:0/3525357616 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6bd01082f0 0x7f6bd01992d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:20:52.783 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:52.778+0000 7f6bcf7fe640 1 --2- 172.21.15.142:0/3525357616 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6bd006b410 0x7f6bd0198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:20:52.783 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:52.778+0000 7f6bc6ffd640 1 -- 172.21.15.142:0/3525357616 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6bd01082f0 msgr2=0x7f6bd01992d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:20:52.783 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:52.778+0000 7f6bc6ffd640 1 --2- 172.21.15.142:0/3525357616 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6bd01082f0 0x7f6bd01992d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:20:52.783 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:52.778+0000 7f6bc6ffd640 1 -- 172.21.15.142:0/3525357616 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6bd006b410 msgr2=0x7f6bd0198040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:20:52.783 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:52.778+0000 7f6bc6ffd640 1 --2- 172.21.15.142:0/3525357616 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6bd006b410 0x7f6bd0198040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:20:52.783 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:52.778+0000 7f6bc6ffd640 1 -- 172.21.15.142:0/3525357616 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bd0199b70 con 0x7f6bd0105740 2024-09-06T14:20:52.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:52.778+0000 7f6bcffff640 1 --2- 172.21.15.142:0/3525357616 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6bd01082f0 0x7f6bd01992d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-06T14:20:52.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:52.778+0000 7f6bcf7fe640 1 --2- 172.21.15.142:0/3525357616 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6bd006b410 0x7f6bd0198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-06T14:20:52.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:52.779+0000 7f6bc6ffd640 1 --2- 172.21.15.142:0/3525357616 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6bd0105740 0x7f6bd0198690 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f6bb8005940 tx=0x7f6bb8032c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:20:52.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:52.779+0000 7f6bcd7fa640 1 -- 172.21.15.142:0/3525357616 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bb803f030 con 0x7f6bd0105740 2024-09-06T14:20:52.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:52.779+0000 7f6bcd7fa640 1 -- 172.21.15.142:0/3525357616 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7f6bb8005480 con 0x7f6bd0105740 2024-09-06T14:20:52.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:52.779+0000 7f6bd4b9e640 1 -- 172.21.15.142:0/3525357616 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6bd0199e00 con 0x7f6bd0105740 2024-09-06T14:20:52.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:52.779+0000 7f6bcd7fa640 1 -- 172.21.15.142:0/3525357616 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bb8035070 con 0x7f6bd0105740 2024-09-06T14:20:52.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:52.779+0000 7f6bd4b9e640 1 -- 172.21.15.142:0/3525357616 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6bd01a0540 con 0x7f6bd0105740 2024-09-06T14:20:52.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:52.780+0000 7f6bcd7fa640 1 -- 172.21.15.142:0/3525357616 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 13) ==== 50132+0+0 (secure 0 0 0) 0x7f6bb8042030 con 0x7f6bd0105740 2024-09-06T14:20:52.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:52.780+0000 7f6bd4b9e640 1 -- 172.21.15.142:0/3525357616 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6bd0105ba0 con 0x7f6bd0105740 2024-09-06T14:20:52.785 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:52.780+0000 7f6bcd7fa640 1 --2- 172.21.15.142:0/3525357616 >> v2:172.21.15.142:6800/853729507 conn(0x7f6bbc03ddc0 0x7f6bbc040280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:20:52.785 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:52.780+0000 7f6bcd7fa640 1 -- 172.21.15.142:0/3525357616 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 983+0+0 (secure 0 0 0) 0x7f6bb8072510 con 0x7f6bd0105740 2024-09-06T14:20:52.785 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:52.780+0000 7f6bcf7fe640 1 --2- 172.21.15.142:0/3525357616 >> v2:172.21.15.142:6800/853729507 conn(0x7f6bbc03ddc0 0x7f6bbc040280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:20:52.785 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:52.781+0000 7f6bcf7fe640 1 --2- 172.21.15.142:0/3525357616 >> v2:172.21.15.142:6800/853729507 conn(0x7f6bbc03ddc0 0x7f6bbc040280 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f6bc0009930 tx=0x7f6bc0008040 comp rx=0 tx=0).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:20:52.785 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:52.784+0000 7f6bcd7fa640 1 -- 172.21.15.142:0/3525357616 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7f6bb8004250 con 0x7f6bd0105740 2024-09-06T14:20:52.796 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Sep 06 14:20:52 smithi162 podman[35581]: 2024-09-06 14:20:52.718414035 +0000 UTC m=+0.113694405 container create 87bdbebbe6a4ddf59589404375c145948a1c2e531f0122bb3af28da2d62727d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c86edd48571ab22373283621eb9feebce9df9c291d4164cfc4df1a09493e1f30, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-x, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-athakkar-testing-2024-09-04-1837-8b38c33, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS) 2024-09-06T14:20:52.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:52.946+0000 7f6bd4b9e640 1 -- 172.21.15.142:0/3525357616 --> v2:172.21.15.142:6800/853729507 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi142:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f6bd0109ec0 con 0x7f6bbc03ddc0 2024-09-06T14:20:53.296 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Sep 06 14:20:52 smithi162 podman[35581]: 2024-09-06 14:20:52.90415245 +0000 UTC m=+0.299432830 container init 87bdbebbe6a4ddf59589404375c145948a1c2e531f0122bb3af28da2d62727d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c86edd48571ab22373283621eb9feebce9df9c291d4164cfc4df1a09493e1f30, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-x, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_CLEAN=True, RELEASE=wip-athakkar-testing-2024-09-04-1837-8b38c33, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-09-06T14:20:53.296 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Sep 06 14:20:52 smithi162 podman[35581]: 2024-09-06 14:20:52.907458239 +0000 UTC m=+0.302738616 container start 87bdbebbe6a4ddf59589404375c145948a1c2e531f0122bb3af28da2d62727d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c86edd48571ab22373283621eb9feebce9df9c291d4164cfc4df1a09493e1f30, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-x, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-athakkar-testing-2024-09-04-1837-8b38c33, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.37.2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-09-06T14:20:53.296 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Sep 06 14:20:52 smithi162 ceph-mgr[35600]: -- 172.21.15.162:0/172838364 <== mon.1 v2:172.21.15.162:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56336ad350e0 con 0x56336ad12800 2024-09-06T14:20:53.296 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Sep 06 14:20:52 smithi162 bash[35581]: 87bdbebbe6a4ddf59589404375c145948a1c2e531f0122bb3af28da2d62727d3 2024-09-06T14:20:53.296 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Sep 06 14:20:52 smithi162 systemd[1]: Started Ceph mgr.x for ce32462e-6c5a-11ef-bce4-c7b262605968. 2024-09-06T14:20:53.957 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:53 smithi142 ceph-mon[29958]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-06T14:20:53.957 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:53 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-06T14:20:53.957 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:53 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-06T14:20:53.957 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:53 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:20:53.957 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:53 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:53.958 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:53 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:53.958 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:53 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:53.958 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:53 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:53.958 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:53 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-06T14:20:53.958 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:53 smithi142 ceph-mon[37372]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-06T14:20:53.958 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:53 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-06T14:20:53.958 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:53 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-06T14:20:53.958 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:53 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:20:53.958 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:53 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:53.958 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:53 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:53.958 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:53 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:53.958 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:53 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:53.959 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:53 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-06T14:20:54.133 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:53 smithi162 ceph-mon[33787]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-06T14:20:54.133 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:53 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-06T14:20:54.133 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:53 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-06T14:20:54.133 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:53 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:20:54.133 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:53 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:54.134 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:53 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:54.134 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:53 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:54.134 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:53 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:54.134 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:53 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-06T14:20:54.546 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Sep 06 14:20:54 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-x[35596]: 2024-09-06T14:20:54.200+0000 7f6aac938040 -1 mgr[py] Module not found: 'volumes' 2024-09-06T14:20:54.546 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Sep 06 14:20:54 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-x[35596]: 2024-09-06T14:20:54.200+0000 7f6aac938040 -1 mgr[py] Traceback (most recent call last): 2024-09-06T14:20:54.546 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Sep 06 14:20:54 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-x[35596]: File "/usr/share/ceph/mgr/volumes/__init__.py", line 2, in 2024-09-06T14:20:54.546 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Sep 06 14:20:54 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-x[35596]: from .module import Module 2024-09-06T14:20:54.546 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Sep 06 14:20:54 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-x[35596]: File "/usr/share/ceph/mgr/volumes/module.py", line 9, in 2024-09-06T14:20:54.547 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Sep 06 14:20:54 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-x[35596]: from .fs.volume import VolumeClient 2024-09-06T14:20:54.547 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Sep 06 14:20:54 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-x[35596]: File "/usr/share/ceph/mgr/volumes/fs/volume.py", line 12, in 2024-09-06T14:20:54.547 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Sep 06 14:20:54 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-x[35596]: from ceph.fs.earmarking import CephFSVolumeEarmarking, EarmarkException # type: ignore 2024-09-06T14:20:54.547 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Sep 06 14:20:54 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-x[35596]: ModuleNotFoundError: No module named 'ceph.fs' 2024-09-06T14:20:54.547 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Sep 06 14:20:54 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-x[35596]: 2024-09-06T14:20:54.547 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Sep 06 14:20:54 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-x[35596]: 2024-09-06T14:20:54.201+0000 7f6aac938040 -1 mgr[py] Class not found in module 'volumes' 2024-09-06T14:20:54.547 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Sep 06 14:20:54 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-x[35596]: 2024-09-06T14:20:54.201+0000 7f6aac938040 -1 mgr[py] Error loading module 'volumes': (2) No such file or directory 2024-09-06T14:20:54.957 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:54 smithi142 ceph-mon[37372]: from='client.14214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi142:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-06T14:20:54.957 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:54 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:54.957 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:54 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:54.957 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:54 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:54.958 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:54 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:20:54.958 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:54 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-06T14:20:54.958 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:54 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:54.958 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:54 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-06T14:20:54.958 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:54 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:20:54.958 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:54 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-06T14:20:54.958 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:54 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:54.958 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:54 smithi142 ceph-mon[29958]: from='client.14214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi142:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-06T14:20:54.958 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:54 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:54.958 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:54 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:54.958 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:54 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:54.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:54 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:20:54.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:54 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-06T14:20:54.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:54 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:54.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:54 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-06T14:20:54.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:54 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:20:54.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:54 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-06T14:20:54.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:54 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:55.295 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:54 smithi162 ceph-mon[33787]: from='client.14214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi142:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-06T14:20:55.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:54 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:55.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:54 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:55.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:54 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:55.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:54 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:20:55.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:54 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-06T14:20:55.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:54 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:55.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:54 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-06T14:20:55.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:54 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:20:55.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:54 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-06T14:20:55.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:54 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:20:55.949 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:55 smithi142 ceph-mon[29958]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-06T14:20:55.949 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:55 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/367924745' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "e7823801-9043-4aaf-a680-046c5b16f18b"} : dispatch 2024-09-06T14:20:55.949 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:55 smithi142 ceph-mon[29958]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "e7823801-9043-4aaf-a680-046c5b16f18b"} : dispatch 2024-09-06T14:20:55.949 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:55 smithi142 ceph-mon[29958]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e7823801-9043-4aaf-a680-046c5b16f18b"}]': finished 2024-09-06T14:20:55.949 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:55 smithi142 ceph-mon[29958]: osdmap e5: 1 total, 0 up, 1 in 2024-09-06T14:20:55.950 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:55 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-06T14:20:55.950 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:55 smithi142 ceph-mon[37372]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-06T14:20:55.950 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:55 smithi142 ceph-mon[37372]: from='client.? 172.21.15.142:0/367924745' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "e7823801-9043-4aaf-a680-046c5b16f18b"} : dispatch 2024-09-06T14:20:55.950 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:55 smithi142 ceph-mon[37372]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "e7823801-9043-4aaf-a680-046c5b16f18b"} : dispatch 2024-09-06T14:20:55.950 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:55 smithi142 ceph-mon[37372]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e7823801-9043-4aaf-a680-046c5b16f18b"}]': finished 2024-09-06T14:20:55.950 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:55 smithi142 ceph-mon[37372]: osdmap e5: 1 total, 0 up, 1 in 2024-09-06T14:20:55.950 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:55 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-06T14:20:56.295 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:55 smithi162 ceph-mon[33787]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-06T14:20:56.295 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:55 smithi162 ceph-mon[33787]: from='client.? 172.21.15.142:0/367924745' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "e7823801-9043-4aaf-a680-046c5b16f18b"} : dispatch 2024-09-06T14:20:56.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:55 smithi162 ceph-mon[33787]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "e7823801-9043-4aaf-a680-046c5b16f18b"} : dispatch 2024-09-06T14:20:56.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:55 smithi162 ceph-mon[33787]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e7823801-9043-4aaf-a680-046c5b16f18b"}]': finished 2024-09-06T14:20:56.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:55 smithi162 ceph-mon[33787]: osdmap e5: 1 total, 0 up, 1 in 2024-09-06T14:20:56.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:55 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-06T14:20:57.207 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:56 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/2355188708' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-06T14:20:57.207 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:56 smithi142 ceph-mon[37372]: from='client.? 172.21.15.142:0/2355188708' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-06T14:20:57.295 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:56 smithi162 ceph-mon[33787]: from='client.? 172.21.15.142:0/2355188708' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-06T14:20:58.207 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:57 smithi142 ceph-mon[29958]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-06T14:20:58.207 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:57 smithi142 ceph-mon[37372]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-06T14:20:58.295 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:57 smithi162 ceph-mon[33787]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-06T14:20:58.296 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Sep 06 14:20:57 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-x[35596]: 2024-09-06T14:20:57.898+0000 7f6aac938040 -1 log_channel(cluster) log [ERR] : Failed to load ceph-mgr modules: volumes 2024-09-06T14:20:58.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:20:58.876+0000 7f6bcd7fa640 1 -- 172.21.15.142:0/3525357616 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mgrmap(e 14) ==== 99730+0+0 (secure 0 0 0) 0x7f6bb8003810 con 0x7f6bd0105740 2024-09-06T14:20:59.207 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:58 smithi142 ceph-mon[29958]: Failed to load ceph-mgr modules: volumes 2024-09-06T14:20:59.207 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:58 smithi142 ceph-mon[29958]: Standby manager daemon x started 2024-09-06T14:20:59.207 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:58 smithi142 ceph-mon[29958]: from='mgr.? 172.21.15.162:0/92948974' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/crt"} : dispatch 2024-09-06T14:20:59.207 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:58 smithi142 ceph-mon[29958]: from='mgr.? 172.21.15.162:0/92948974' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-09-06T14:20:59.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:58 smithi142 ceph-mon[29958]: from='mgr.? 172.21.15.162:0/92948974' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/key"} : dispatch 2024-09-06T14:20:59.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:58 smithi142 ceph-mon[29958]: from='mgr.? 172.21.15.162:0/92948974' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-09-06T14:20:59.208 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:58 smithi142 ceph-mon[37372]: Failed to load ceph-mgr modules: volumes 2024-09-06T14:20:59.208 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:58 smithi142 ceph-mon[37372]: Standby manager daemon x started 2024-09-06T14:20:59.208 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:58 smithi142 ceph-mon[37372]: from='mgr.? 172.21.15.162:0/92948974' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/crt"} : dispatch 2024-09-06T14:20:59.208 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:58 smithi142 ceph-mon[37372]: from='mgr.? 172.21.15.162:0/92948974' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-09-06T14:20:59.208 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:58 smithi142 ceph-mon[37372]: from='mgr.? 172.21.15.162:0/92948974' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/key"} : dispatch 2024-09-06T14:20:59.208 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:58 smithi142 ceph-mon[37372]: from='mgr.? 172.21.15.162:0/92948974' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-09-06T14:20:59.295 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:58 smithi162 ceph-mon[33787]: Failed to load ceph-mgr modules: volumes 2024-09-06T14:20:59.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:58 smithi162 ceph-mon[33787]: Standby manager daemon x started 2024-09-06T14:20:59.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:58 smithi162 ceph-mon[33787]: from='mgr.? 172.21.15.162:0/92948974' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/crt"} : dispatch 2024-09-06T14:20:59.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:58 smithi162 ceph-mon[33787]: from='mgr.? 172.21.15.162:0/92948974' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-09-06T14:20:59.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:58 smithi162 ceph-mon[33787]: from='mgr.? 172.21.15.162:0/92948974' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/key"} : dispatch 2024-09-06T14:20:59.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:58 smithi162 ceph-mon[33787]: from='mgr.? 172.21.15.162:0/92948974' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-09-06T14:20:59.957 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:59 smithi142 ceph-mon[29958]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-06T14:20:59.957 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:59 smithi142 ceph-mon[29958]: mgrmap e14: y(active, since 60s), standbys: x 2024-09-06T14:20:59.957 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:20:59 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-06T14:20:59.958 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:59 smithi142 ceph-mon[37372]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-06T14:20:59.958 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:59 smithi142 ceph-mon[37372]: mgrmap e14: y(active, since 60s), standbys: x 2024-09-06T14:20:59.958 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:20:59 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-06T14:21:00.295 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:59 smithi162 ceph-mon[33787]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-06T14:21:00.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:59 smithi162 ceph-mon[33787]: mgrmap e14: y(active, since 60s), standbys: x 2024-09-06T14:21:00.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:20:59 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-06T14:21:01.957 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:01 smithi142 ceph-mon[29958]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-06T14:21:01.958 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:01 smithi142 ceph-mon[37372]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-06T14:21:02.295 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:01 smithi162 ceph-mon[33787]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-06T14:21:03.133 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:02 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-06T14:21:03.133 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:02 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:21:03.134 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:02 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-06T14:21:03.134 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:02 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:21:03.295 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:02 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-06T14:21:03.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:02 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:21:04.088 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:03 smithi142 ceph-mon[29958]: Deploying daemon osd.0 on smithi142 2024-09-06T14:21:04.089 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:03 smithi142 ceph-mon[29958]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-06T14:21:04.089 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:03 smithi142 ceph-mon[37372]: Deploying daemon osd.0 on smithi142 2024-09-06T14:21:04.089 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:03 smithi142 ceph-mon[37372]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-06T14:21:04.295 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:03 smithi162 ceph-mon[33787]: Deploying daemon osd.0 on smithi142 2024-09-06T14:21:04.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:03 smithi162 ceph-mon[33787]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-06T14:21:05.957 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:05 smithi142 ceph-mon[29958]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-06T14:21:05.958 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:05 smithi142 ceph-mon[37372]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-06T14:21:06.295 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:05 smithi162 ceph-mon[33787]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-06T14:21:07.958 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:07 smithi142 ceph-mon[29958]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-06T14:21:07.958 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:07 smithi142 ceph-mon[37372]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-06T14:21:08.295 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:07 smithi162 ceph-mon[33787]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-06T14:21:09.957 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:09 smithi142 ceph-mon[29958]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-06T14:21:09.958 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:09 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-06T14:21:09.958 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:09 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:21:09.958 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:09 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:21:09.958 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:09 smithi142 ceph-mon[37372]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-06T14:21:09.958 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:09 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-06T14:21:09.958 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:09 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:21:09.958 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:09 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:21:10.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:09 smithi162 ceph-mon[33787]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-06T14:21:10.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:09 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-06T14:21:10.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:09 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:21:10.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:09 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:21:12.207 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:11 smithi142 ceph-mon[29958]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-06T14:21:12.207 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:11 smithi142 ceph-mon[37372]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-06T14:21:12.295 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:11 smithi162 ceph-mon[33787]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-06T14:21:13.162 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:12 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:21:13.163 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:12 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:21:13.163 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:12 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:21:13.163 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:12 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-06T14:21:13.163 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:12 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:21:13.163 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:12 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:21:13.163 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:12 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:21:13.163 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:12 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:21:13.163 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:12 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-06T14:21:13.163 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:12 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:21:13.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:12 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:21:13.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:12 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:21:13.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:12 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:21:13.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:12 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-06T14:21:13.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:12 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:21:13.493 INFO:teuthology.orchestra.run.smithi142.stdout:Created osd(s) 0 on host 'smithi142' 2024-09-06T14:21:13.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:13.498+0000 7f6bcd7fa640 1 -- 172.21.15.142:0/3525357616 <== mgr.14146 v2:172.21.15.142:6800/853729507 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f6bd0109ec0 con 0x7f6bbc03ddc0 2024-09-06T14:21:13.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:13.500+0000 7f6bd4b9e640 1 -- 172.21.15.142:0/3525357616 >> v2:172.21.15.142:6800/853729507 conn(0x7f6bbc03ddc0 msgr2=0x7f6bbc040280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:21:13.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:13.500+0000 7f6bd4b9e640 1 --2- 172.21.15.142:0/3525357616 >> v2:172.21.15.142:6800/853729507 conn(0x7f6bbc03ddc0 0x7f6bbc040280 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f6bc0009930 tx=0x7f6bc0008040 comp rx=0 tx=0).stop 2024-09-06T14:21:13.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:13.500+0000 7f6bd4b9e640 1 -- 172.21.15.142:0/3525357616 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6bd0105740 msgr2=0x7f6bd0198690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:21:13.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:13.500+0000 7f6bd4b9e640 1 --2- 172.21.15.142:0/3525357616 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6bd0105740 0x7f6bd0198690 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f6bb8005940 tx=0x7f6bb8032c70 comp rx=0 tx=0).stop 2024-09-06T14:21:13.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:13.500+0000 7f6bd4b9e640 1 -- 172.21.15.142:0/3525357616 shutdown_connections 2024-09-06T14:21:13.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:13.500+0000 7f6bd4b9e640 1 --2- 172.21.15.142:0/3525357616 >> v2:172.21.15.142:6800/853729507 conn(0x7f6bbc03ddc0 0x7f6bbc040280 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:21:13.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:13.500+0000 7f6bd4b9e640 1 --2- 172.21.15.142:0/3525357616 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6bd01082f0 0x7f6bd01992d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:21:13.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:13.500+0000 7f6bd4b9e640 1 --2- 172.21.15.142:0/3525357616 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6bd0105740 0x7f6bd0198690 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:21:13.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:13.500+0000 7f6bd4b9e640 1 --2- 172.21.15.142:0/3525357616 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6bd006b410 0x7f6bd0198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:21:13.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:13.500+0000 7f6bd4b9e640 1 -- 172.21.15.142:0/3525357616 >> 172.21.15.142:0/3525357616 conn(0x7f6bd00fcd80 msgr2=0x7f6bd01087a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:21:13.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:13.500+0000 7f6bd4b9e640 1 -- 172.21.15.142:0/3525357616 shutdown_connections 2024-09-06T14:21:13.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:13.500+0000 7f6bd4b9e640 1 -- 172.21.15.142:0/3525357616 wait complete. 2024-09-06T14:21:14.117 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:13 smithi142 ceph-mon[29958]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-06T14:21:14.117 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:13 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-06T14:21:14.117 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:13 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:21:14.117 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:13 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:21:14.117 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:13 smithi142 ceph-mon[37372]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-06T14:21:14.118 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:13 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-06T14:21:14.118 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:13 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:21:14.118 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:13 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:21:14.149 DEBUG:teuthology.orchestra.run.smithi142:osd.0> sudo journalctl -f -n 0 -u ceph-ce32462e-6c5a-11ef-bce4-c7b262605968@osd.0.service 2024-09-06T14:21:14.151 INFO:tasks.cephadm:Deploying osd.1 on smithi142 with /dev/vg_nvme/lv_3... 2024-09-06T14:21:14.152 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- lvm zap /dev/vg_nvme/lv_3 2024-09-06T14:21:14.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:13 smithi162 ceph-mon[33787]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-06T14:21:14.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:13 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-06T14:21:14.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:13 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:21:14.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:13 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:21:15.207 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:14 smithi142 ceph-mon[29958]: from='osd.0 v2:172.21.15.142:6801/1649056378' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-06T14:21:15.208 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:14 smithi142 ceph-mon[37372]: from='osd.0 v2:172.21.15.142:6801/1649056378' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-06T14:21:15.295 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:14 smithi162 ceph-mon[33787]: from='osd.0 v2:172.21.15.142:6801/1649056378' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-06T14:21:15.351 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:21:16.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:15 smithi142 ceph-mon[37372]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-06T14:21:16.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:15 smithi142 ceph-mon[37372]: from='osd.0 v2:172.21.15.142:6801/1649056378' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-06T14:21:16.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:15 smithi142 ceph-mon[37372]: osdmap e6: 1 total, 0 up, 1 in 2024-09-06T14:21:16.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:15 smithi142 ceph-mon[37372]: from='osd.0 v2:172.21.15.142:6801/1649056378' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi142", "root=default"]} : dispatch 2024-09-06T14:21:16.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:15 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-06T14:21:16.210 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:15 smithi142 ceph-mon[29958]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-06T14:21:16.210 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:15 smithi142 ceph-mon[29958]: from='osd.0 v2:172.21.15.142:6801/1649056378' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-06T14:21:16.210 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:15 smithi142 ceph-mon[29958]: osdmap e6: 1 total, 0 up, 1 in 2024-09-06T14:21:16.210 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:15 smithi142 ceph-mon[29958]: from='osd.0 v2:172.21.15.142:6801/1649056378' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi142", "root=default"]} : dispatch 2024-09-06T14:21:16.210 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:15 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-06T14:21:16.210 INFO:journalctl@ceph.osd.0.smithi142.stdout:Sep 06 14:21:15 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-0[41141]: 2024-09-06T14:21:15.919+0000 7f66d14c8640 -1 osd.0 0 waiting for initial osdmap 2024-09-06T14:21:16.210 INFO:journalctl@ceph.osd.0.smithi142.stdout:Sep 06 14:21:15 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-0[41141]: 2024-09-06T14:21:15.921+0000 7f66cd2ef640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-06T14:21:16.295 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:15 smithi162 ceph-mon[33787]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-06T14:21:16.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:15 smithi162 ceph-mon[33787]: from='osd.0 v2:172.21.15.142:6801/1649056378' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-06T14:21:16.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:15 smithi162 ceph-mon[33787]: osdmap e6: 1 total, 0 up, 1 in 2024-09-06T14:21:16.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:15 smithi162 ceph-mon[33787]: from='osd.0 v2:172.21.15.142:6801/1649056378' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi142", "root=default"]} : dispatch 2024-09-06T14:21:16.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:15 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-06T14:21:17.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:16 smithi142 ceph-mon[29958]: from='osd.0 v2:172.21.15.142:6801/1649056378' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]': finished 2024-09-06T14:21:17.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:16 smithi142 ceph-mon[29958]: osdmap e7: 1 total, 0 up, 1 in 2024-09-06T14:21:17.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:16 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-06T14:21:17.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:16 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-06T14:21:17.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:16 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-06T14:21:17.208 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:16 smithi142 ceph-mon[37372]: from='osd.0 v2:172.21.15.142:6801/1649056378' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]': finished 2024-09-06T14:21:17.208 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:16 smithi142 ceph-mon[37372]: osdmap e7: 1 total, 0 up, 1 in 2024-09-06T14:21:17.208 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:16 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-06T14:21:17.208 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:16 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-06T14:21:17.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:16 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-06T14:21:17.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:16 smithi162 ceph-mon[33787]: from='osd.0 v2:172.21.15.142:6801/1649056378' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]': finished 2024-09-06T14:21:17.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:16 smithi162 ceph-mon[33787]: osdmap e7: 1 total, 0 up, 1 in 2024-09-06T14:21:17.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:16 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-06T14:21:17.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:16 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-06T14:21:17.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:16 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-06T14:21:18.091 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:17 smithi142 ceph-mon[29958]: purged_snaps scrub starts 2024-09-06T14:21:18.092 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:17 smithi142 ceph-mon[29958]: purged_snaps scrub ok 2024-09-06T14:21:18.092 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:17 smithi142 ceph-mon[29958]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-06T14:21:18.092 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:17 smithi142 ceph-mon[29958]: osd.0 v2:172.21.15.142:6801/1649056378 boot 2024-09-06T14:21:18.092 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:17 smithi142 ceph-mon[29958]: osdmap e8: 1 total, 1 up, 1 in 2024-09-06T14:21:18.092 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:17 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-06T14:21:18.092 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:17 smithi142 ceph-mon[37372]: purged_snaps scrub starts 2024-09-06T14:21:18.092 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:17 smithi142 ceph-mon[37372]: purged_snaps scrub ok 2024-09-06T14:21:18.092 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:17 smithi142 ceph-mon[37372]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-06T14:21:18.093 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:17 smithi142 ceph-mon[37372]: osd.0 v2:172.21.15.142:6801/1649056378 boot 2024-09-06T14:21:18.093 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:17 smithi142 ceph-mon[37372]: osdmap e8: 1 total, 1 up, 1 in 2024-09-06T14:21:18.093 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:17 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-06T14:21:18.295 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:17 smithi162 ceph-mon[33787]: purged_snaps scrub starts 2024-09-06T14:21:18.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:17 smithi162 ceph-mon[33787]: purged_snaps scrub ok 2024-09-06T14:21:18.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:17 smithi162 ceph-mon[33787]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-06T14:21:18.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:17 smithi162 ceph-mon[33787]: osd.0 v2:172.21.15.142:6801/1649056378 boot 2024-09-06T14:21:18.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:17 smithi162 ceph-mon[33787]: osdmap e8: 1 total, 1 up, 1 in 2024-09-06T14:21:18.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:17 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-06T14:21:19.207 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:18 smithi142 ceph-mon[29958]: osdmap e9: 1 total, 1 up, 1 in 2024-09-06T14:21:19.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:18 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:21:19.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:18 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:21:19.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:18 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-06T14:21:19.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:18 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:21:19.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:18 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:21:19.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:18 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-06T14:21:19.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:18 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:21:19.208 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:18 smithi142 ceph-mon[37372]: osdmap e9: 1 total, 1 up, 1 in 2024-09-06T14:21:19.208 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:18 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:21:19.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:18 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:21:19.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:18 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-06T14:21:19.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:18 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:21:19.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:18 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:21:19.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:18 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-06T14:21:19.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:18 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:21:19.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:18 smithi162 ceph-mon[33787]: osdmap e9: 1 total, 1 up, 1 in 2024-09-06T14:21:19.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:18 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:21:19.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:18 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:21:19.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:18 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-06T14:21:19.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:18 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:21:19.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:18 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:21:19.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:18 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-06T14:21:19.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:18 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:21:19.551 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:21:19.575 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph orch daemon add osd smithi142:vg_nvme/lv_3 2024-09-06T14:21:19.819 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:21:20.193 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:19 smithi142 ceph-mon[29958]: Detected new or changed devices on smithi142 2024-09-06T14:21:20.193 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:19 smithi142 ceph-mon[29958]: Adjusting osd_memory_target on smithi142 to 16064M 2024-09-06T14:21:20.193 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:19 smithi142 ceph-mon[29958]: pgmap v39: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-06T14:21:20.194 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:19 smithi142 ceph-mon[37372]: Detected new or changed devices on smithi142 2024-09-06T14:21:20.194 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:19 smithi142 ceph-mon[37372]: Adjusting osd_memory_target on smithi142 to 16064M 2024-09-06T14:21:20.194 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:19 smithi142 ceph-mon[37372]: pgmap v39: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-06T14:21:20.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:20.281+0000 7f495d917640 1 -- 172.21.15.142:0/4175420690 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4958102760 msgr2=0x7f4958102b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:21:20.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:20.281+0000 7f495d917640 1 --2- 172.21.15.142:0/4175420690 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4958102760 0x7f4958102b60 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f4948009930 tx=0x7f494802a420 comp rx=0 tx=0).stop 2024-09-06T14:21:20.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:20.281+0000 7f495d917640 1 -- 172.21.15.142:0/4175420690 shutdown_connections 2024-09-06T14:21:20.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:20.281+0000 7f495d917640 1 --2- 172.21.15.142:0/4175420690 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4958104280 0x7f495810ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:21:20.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:20.281+0000 7f495d917640 1 --2- 172.21.15.142:0/4175420690 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f49581038c0 0x7f4958103d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:21:20.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:20.281+0000 7f495d917640 1 --2- 172.21.15.142:0/4175420690 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4958102760 0x7f4958102b60 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:21:20.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:20.281+0000 7f495d917640 1 -- 172.21.15.142:0/4175420690 >> 172.21.15.142:0/4175420690 conn(0x7f49580fe050 msgr2=0x7f4958100470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:21:20.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:20.281+0000 7f495d917640 1 -- 172.21.15.142:0/4175420690 shutdown_connections 2024-09-06T14:21:20.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:20.281+0000 7f495d917640 1 -- 172.21.15.142:0/4175420690 wait complete. 2024-09-06T14:21:20.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:20.281+0000 7f495d917640 1 Processor -- start 2024-09-06T14:21:20.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:20.282+0000 7f495d917640 1 -- start start 2024-09-06T14:21:20.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:20.282+0000 7f495d917640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4958102760 0x7f4958076a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:21:20.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:20.282+0000 7f495d917640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f49581038c0 0x7f49580752a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:21:20.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:20.282+0000 7f495d917640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4958104280 0x7f49580757e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:21:20.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:20.282+0000 7f495d917640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f495810dac0 con 0x7f49581038c0 2024-09-06T14:21:20.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:20.282+0000 7f495d917640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f495810d940 con 0x7f4958104280 2024-09-06T14:21:20.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:20.282+0000 7f495d917640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7f495810dc40 con 0x7f4958102760 2024-09-06T14:21:20.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:20.282+0000 7f4956ffd640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4958102760 0x7f4958076a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:21:20.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:20.282+0000 7f49567fc640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f49581038c0 0x7f49580752a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:21:20.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:20.282+0000 7f4956ffd640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4958102760 0x7f4958076a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:50848/0 (socket says 172.21.15.142:50848) 2024-09-06T14:21:20.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:20.282+0000 7f49567fc640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f49581038c0 0x7f49580752a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:40532/0 (socket says 172.21.15.142:40532) 2024-09-06T14:21:20.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:20.282+0000 7f4956ffd640 1 -- 172.21.15.142:0/3747008207 learned_addr learned my addr 172.21.15.142:0/3747008207 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:21:20.295 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:20.282+0000 7f49577fe640 1 --2- 172.21.15.142:0/3747008207 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4958104280 0x7f49580757e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:21:20.295 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:20.282+0000 7f4956ffd640 1 -- 172.21.15.142:0/3747008207 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4958104280 msgr2=0x7f49580757e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:21:20.295 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:20.282+0000 7f4956ffd640 1 --2- 172.21.15.142:0/3747008207 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4958104280 0x7f49580757e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:21:20.295 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:20.282+0000 7f4956ffd640 1 -- 172.21.15.142:0/3747008207 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f49581038c0 msgr2=0x7f49580752a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:21:20.295 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:20.282+0000 7f4956ffd640 1 --2- 172.21.15.142:0/3747008207 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f49581038c0 0x7f49580752a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:21:20.295 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:20.282+0000 7f4956ffd640 1 -- 172.21.15.142:0/3747008207 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49580760a0 con 0x7f4958102760 2024-09-06T14:21:20.295 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:20.282+0000 7f49567fc640 1 --2- 172.21.15.142:0/3747008207 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f49581038c0 0x7f49580752a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-06T14:21:20.295 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:20.282+0000 7f49577fe640 1 --2- 172.21.15.142:0/3747008207 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4958104280 0x7f49580757e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-06T14:21:20.295 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:20.282+0000 7f4956ffd640 1 --2- 172.21.15.142:0/3747008207 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4958102760 0x7f4958076a30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4948005ec0 tx=0x7f4948003c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:21:20.295 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:20.283+0000 7f495c915640 1 -- 172.21.15.142:0/3747008207 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f494803f070 con 0x7f4958102760 2024-09-06T14:21:20.295 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:20.283+0000 7f495d917640 1 -- 172.21.15.142:0/3747008207 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49581a22d0 con 0x7f4958102760 2024-09-06T14:21:20.296 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:20.283+0000 7f495c915640 1 -- 172.21.15.142:0/3747008207 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7f494803bcf0 con 0x7f4958102760 2024-09-06T14:21:20.296 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:20.283+0000 7f495c915640 1 -- 172.21.15.142:0/3747008207 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4948035030 con 0x7f4958102760 2024-09-06T14:21:20.296 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:20.283+0000 7f495d917640 1 -- 172.21.15.142:0/3747008207 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f49581a27d0 con 0x7f4958102760 2024-09-06T14:21:20.296 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:20.284+0000 7f495d917640 1 -- 172.21.15.142:0/3747008207 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4958102b60 con 0x7f4958102760 2024-09-06T14:21:20.296 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:20.284+0000 7f495c915640 1 -- 172.21.15.142:0/3747008207 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 14) ==== 99730+0+0 (secure 0 0 0) 0x7f4948042070 con 0x7f4958102760 2024-09-06T14:21:20.296 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:20.285+0000 7f495c915640 1 --2- 172.21.15.142:0/3747008207 >> v2:172.21.15.142:6800/853729507 conn(0x7f49400774a0 0x7f4940079960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:21:20.296 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:20.285+0000 7f49567fc640 1 --2- 172.21.15.142:0/3747008207 >> v2:172.21.15.142:6800/853729507 conn(0x7f49400774a0 0x7f4940079960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:21:20.296 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:20.285+0000 7f495c915640 1 -- 172.21.15.142:0/3747008207 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(9..9 src has 1..9) ==== 1455+0+0 (secure 0 0 0) 0x7f49480b8c50 con 0x7f4958102760 2024-09-06T14:21:20.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:20.285+0000 7f49567fc640 1 --2- 172.21.15.142:0/3747008207 >> v2:172.21.15.142:6800/853729507 conn(0x7f49400774a0 0x7f4940079960 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f494c0095f0 tx=0x7f494c005f90 comp rx=0 tx=0).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:21:20.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:20.289+0000 7f495c915640 1 -- 172.21.15.142:0/3747008207 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7f49480be020 con 0x7f4958102760 2024-09-06T14:21:20.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:19 smithi162 ceph-mon[33787]: Detected new or changed devices on smithi142 2024-09-06T14:21:20.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:19 smithi162 ceph-mon[33787]: Adjusting osd_memory_target on smithi142 to 16064M 2024-09-06T14:21:20.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:19 smithi162 ceph-mon[33787]: pgmap v39: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-06T14:21:20.445 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:20.450+0000 7f495d917640 1 -- 172.21.15.142:0/3747008207 --> v2:172.21.15.142:6800/853729507 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi142:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f495802e5f0 con 0x7f49400774a0 2024-09-06T14:21:21.100 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:20 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-06T14:21:21.100 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:20 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-06T14:21:21.101 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:20 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:21:21.101 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:20 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-06T14:21:21.101 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:20 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-06T14:21:21.101 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:20 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:21:21.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:20 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-06T14:21:21.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:20 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-06T14:21:21.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:20 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:21:22.117 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:21 smithi142 ceph-mon[29958]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi142:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-06T14:21:22.117 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:21 smithi142 ceph-mon[29958]: pgmap v40: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-06T14:21:22.117 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:21 smithi142 ceph-mon[37372]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi142:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-06T14:21:22.118 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:21 smithi142 ceph-mon[37372]: pgmap v40: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-06T14:21:22.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:21 smithi162 ceph-mon[33787]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi142:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-06T14:21:22.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:21 smithi162 ceph-mon[33787]: pgmap v40: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-06T14:21:23.207 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:22 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/3092694892' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "364b1ffb-bdda-4b70-bd6c-3bcb73110359"} : dispatch 2024-09-06T14:21:23.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:22 smithi142 ceph-mon[29958]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "364b1ffb-bdda-4b70-bd6c-3bcb73110359"} : dispatch 2024-09-06T14:21:23.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:22 smithi142 ceph-mon[29958]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "364b1ffb-bdda-4b70-bd6c-3bcb73110359"}]': finished 2024-09-06T14:21:23.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:22 smithi142 ceph-mon[29958]: osdmap e10: 2 total, 1 up, 2 in 2024-09-06T14:21:23.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:22 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-06T14:21:23.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:22 smithi142 ceph-mon[29958]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-06T14:21:23.208 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:22 smithi142 ceph-mon[37372]: from='client.? 172.21.15.142:0/3092694892' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "364b1ffb-bdda-4b70-bd6c-3bcb73110359"} : dispatch 2024-09-06T14:21:23.208 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:22 smithi142 ceph-mon[37372]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "364b1ffb-bdda-4b70-bd6c-3bcb73110359"} : dispatch 2024-09-06T14:21:23.208 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:22 smithi142 ceph-mon[37372]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "364b1ffb-bdda-4b70-bd6c-3bcb73110359"}]': finished 2024-09-06T14:21:23.208 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:22 smithi142 ceph-mon[37372]: osdmap e10: 2 total, 1 up, 2 in 2024-09-06T14:21:23.208 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:22 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-06T14:21:23.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:22 smithi142 ceph-mon[37372]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-06T14:21:23.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:22 smithi162 ceph-mon[33787]: from='client.? 172.21.15.142:0/3092694892' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "364b1ffb-bdda-4b70-bd6c-3bcb73110359"} : dispatch 2024-09-06T14:21:23.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:22 smithi162 ceph-mon[33787]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "364b1ffb-bdda-4b70-bd6c-3bcb73110359"} : dispatch 2024-09-06T14:21:23.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:22 smithi162 ceph-mon[33787]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "364b1ffb-bdda-4b70-bd6c-3bcb73110359"}]': finished 2024-09-06T14:21:23.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:22 smithi162 ceph-mon[33787]: osdmap e10: 2 total, 1 up, 2 in 2024-09-06T14:21:23.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:22 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-06T14:21:23.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:22 smithi162 ceph-mon[33787]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-06T14:21:24.207 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:23 smithi142 ceph-mon[37372]: from='client.? 172.21.15.142:0/3114598069' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-06T14:21:24.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:23 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/3114598069' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-06T14:21:24.295 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:23 smithi162 ceph-mon[33787]: from='client.? 172.21.15.142:0/3114598069' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-06T14:21:25.207 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:24 smithi142 ceph-mon[29958]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-06T14:21:25.207 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:24 smithi142 ceph-mon[37372]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-06T14:21:25.295 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:24 smithi162 ceph-mon[33787]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-06T14:21:28.207 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:27 smithi142 ceph-mon[29958]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-06T14:21:28.207 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:27 smithi142 ceph-mon[37372]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-06T14:21:28.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:27 smithi162 ceph-mon[33787]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-06T14:21:30.119 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:29 smithi142 ceph-mon[29958]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-06T14:21:30.119 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:29 smithi142 ceph-mon[37372]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-06T14:21:30.295 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:29 smithi162 ceph-mon[33787]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-06T14:21:31.207 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:30 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-06T14:21:31.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:30 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:21:31.208 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:30 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-06T14:21:31.208 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:30 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:21:31.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:30 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-06T14:21:31.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:30 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:21:32.120 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:31 smithi142 ceph-mon[29958]: Deploying daemon osd.1 on smithi142 2024-09-06T14:21:32.120 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:31 smithi142 ceph-mon[29958]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-06T14:21:32.120 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:31 smithi142 ceph-mon[37372]: Deploying daemon osd.1 on smithi142 2024-09-06T14:21:32.120 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:31 smithi142 ceph-mon[37372]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-06T14:21:32.295 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:31 smithi162 ceph-mon[33787]: Deploying daemon osd.1 on smithi142 2024-09-06T14:21:32.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:31 smithi162 ceph-mon[33787]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-06T14:21:34.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:33 smithi142 ceph-mon[29958]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-06T14:21:34.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:33 smithi142 ceph-mon[37372]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-06T14:21:34.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:33 smithi162 ceph-mon[33787]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-06T14:21:36.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:35 smithi142 ceph-mon[29958]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-06T14:21:36.208 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:35 smithi142 ceph-mon[37372]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-06T14:21:36.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:35 smithi162 ceph-mon[33787]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-06T14:21:37.196 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:36 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-06T14:21:37.196 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:36 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:21:37.196 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:36 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:21:37.196 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:36 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-06T14:21:37.196 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:36 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:21:37.196 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:36 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:21:37.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:36 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-06T14:21:37.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:36 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:21:37.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:36 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:21:38.158 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:37 smithi142 ceph-mon[29958]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-06T14:21:38.158 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:37 smithi142 ceph-mon[37372]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-06T14:21:38.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:37 smithi162 ceph-mon[33787]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-06T14:21:40.208 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:39 smithi142 ceph-mon[37372]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-06T14:21:40.208 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:39 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:21:40.208 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:39 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:21:40.208 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:39 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:21:40.208 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:39 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-06T14:21:40.208 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:39 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:21:40.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:39 smithi142 ceph-mon[29958]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-06T14:21:40.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:39 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:21:40.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:39 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:21:40.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:39 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:21:40.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:39 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-06T14:21:40.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:39 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:21:40.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:39 smithi162 ceph-mon[33787]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-06T14:21:40.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:39 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:21:40.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:39 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:21:40.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:39 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:21:40.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:39 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-06T14:21:40.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:39 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:21:40.616 INFO:teuthology.orchestra.run.smithi142.stdout:Created osd(s) 1 on host 'smithi142' 2024-09-06T14:21:40.616 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:40.620+0000 7f495c915640 1 -- 172.21.15.142:0/3747008207 <== mgr.14146 v2:172.21.15.142:6800/853729507 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f495802e5f0 con 0x7f49400774a0 2024-09-06T14:21:40.619 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:40.623+0000 7f495d917640 1 -- 172.21.15.142:0/3747008207 >> v2:172.21.15.142:6800/853729507 conn(0x7f49400774a0 msgr2=0x7f4940079960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:21:40.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:40.623+0000 7f495d917640 1 --2- 172.21.15.142:0/3747008207 >> v2:172.21.15.142:6800/853729507 conn(0x7f49400774a0 0x7f4940079960 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f494c0095f0 tx=0x7f494c005f90 comp rx=0 tx=0).stop 2024-09-06T14:21:40.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:40.623+0000 7f495d917640 1 -- 172.21.15.142:0/3747008207 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4958102760 msgr2=0x7f4958076a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:21:40.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:40.623+0000 7f495d917640 1 --2- 172.21.15.142:0/3747008207 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4958102760 0x7f4958076a30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4948005ec0 tx=0x7f4948003c80 comp rx=0 tx=0).stop 2024-09-06T14:21:40.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:40.623+0000 7f495d917640 1 -- 172.21.15.142:0/3747008207 shutdown_connections 2024-09-06T14:21:40.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:40.623+0000 7f495d917640 1 --2- 172.21.15.142:0/3747008207 >> v2:172.21.15.142:6800/853729507 conn(0x7f49400774a0 0x7f4940079960 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:21:40.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:40.623+0000 7f495d917640 1 --2- 172.21.15.142:0/3747008207 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4958104280 0x7f49580757e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:21:40.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:40.623+0000 7f495d917640 1 --2- 172.21.15.142:0/3747008207 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f49581038c0 0x7f49580752a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:21:40.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:40.623+0000 7f495d917640 1 --2- 172.21.15.142:0/3747008207 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4958102760 0x7f4958076a30 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:21:40.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:40.623+0000 7f495d917640 1 -- 172.21.15.142:0/3747008207 >> 172.21.15.142:0/3747008207 conn(0x7f49580fe050 msgr2=0x7f49580ffbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:21:40.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:40.623+0000 7f495d917640 1 -- 172.21.15.142:0/3747008207 shutdown_connections 2024-09-06T14:21:40.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:40.623+0000 7f495d917640 1 -- 172.21.15.142:0/3747008207 wait complete. 2024-09-06T14:21:41.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:40 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-06T14:21:41.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:40 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:21:41.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:40 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:21:41.208 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:40 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-06T14:21:41.208 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:40 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:21:41.208 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:40 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:21:41.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:40 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-06T14:21:41.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:40 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:21:41.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:40 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:21:41.297 DEBUG:teuthology.orchestra.run.smithi142:osd.1> sudo journalctl -f -n 0 -u ceph-ce32462e-6c5a-11ef-bce4-c7b262605968@osd.1.service 2024-09-06T14:21:41.300 INFO:tasks.cephadm:Deploying osd.2 on smithi142 with /dev/vg_nvme/lv_2... 2024-09-06T14:21:41.300 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- lvm zap /dev/vg_nvme/lv_2 2024-09-06T14:21:41.932 INFO:journalctl@ceph.osd.1.smithi142.stdout:Sep 06 14:21:41 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-1[47186]: 2024-09-06T14:21:41.735+0000 7ff2d4162640 -1 osd.1 0 log_to_monitors true 2024-09-06T14:21:42.073 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:21:42.193 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:41 smithi142 ceph-mon[29958]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-06T14:21:42.193 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:41 smithi142 ceph-mon[29958]: from='osd.1 v2:172.21.15.142:6805/599802221' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-06T14:21:42.193 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:41 smithi142 ceph-mon[37372]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-06T14:21:42.194 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:41 smithi142 ceph-mon[37372]: from='osd.1 v2:172.21.15.142:6805/599802221' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-06T14:21:42.197 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:41 smithi162 ceph-mon[33787]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-06T14:21:42.197 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:41 smithi162 ceph-mon[33787]: from='osd.1 v2:172.21.15.142:6805/599802221' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-06T14:21:43.796 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:43 smithi162 ceph-mon[33787]: from='osd.1 v2:172.21.15.142:6805/599802221' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-06T14:21:43.796 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:43 smithi162 ceph-mon[33787]: osdmap e11: 2 total, 1 up, 2 in 2024-09-06T14:21:43.796 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:43 smithi162 ceph-mon[33787]: from='osd.1 v2:172.21.15.142:6805/599802221' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi142", "root=default"]} : dispatch 2024-09-06T14:21:43.796 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:43 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-06T14:21:43.796 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:43 smithi162 ceph-mon[33787]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-06T14:21:43.856 INFO:journalctl@ceph.osd.1.smithi142.stdout:Sep 06 14:21:43 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-1[47186]: 2024-09-06T14:21:43.628+0000 7ff2d00e5640 -1 osd.1 0 waiting for initial osdmap 2024-09-06T14:21:43.856 INFO:journalctl@ceph.osd.1.smithi142.stdout:Sep 06 14:21:43 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-1[47186]: 2024-09-06T14:21:43.632+0000 7ff2cbf0c640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-06T14:21:43.856 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:43 smithi142 ceph-mon[29958]: from='osd.1 v2:172.21.15.142:6805/599802221' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-06T14:21:43.857 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:43 smithi142 ceph-mon[29958]: osdmap e11: 2 total, 1 up, 2 in 2024-09-06T14:21:43.857 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:43 smithi142 ceph-mon[29958]: from='osd.1 v2:172.21.15.142:6805/599802221' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi142", "root=default"]} : dispatch 2024-09-06T14:21:43.857 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:43 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-06T14:21:43.857 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:43 smithi142 ceph-mon[29958]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-06T14:21:43.857 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:43 smithi142 ceph-mon[37372]: from='osd.1 v2:172.21.15.142:6805/599802221' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-06T14:21:43.857 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:43 smithi142 ceph-mon[37372]: osdmap e11: 2 total, 1 up, 2 in 2024-09-06T14:21:43.857 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:43 smithi142 ceph-mon[37372]: from='osd.1 v2:172.21.15.142:6805/599802221' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi142", "root=default"]} : dispatch 2024-09-06T14:21:43.857 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:43 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-06T14:21:43.857 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:43 smithi142 ceph-mon[37372]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-06T14:21:44.958 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:44 smithi142 ceph-mon[37372]: from='osd.1 v2:172.21.15.142:6805/599802221' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]': finished 2024-09-06T14:21:44.958 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:44 smithi142 ceph-mon[37372]: osdmap e12: 2 total, 1 up, 2 in 2024-09-06T14:21:44.958 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:44 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-06T14:21:44.958 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:44 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-06T14:21:44.958 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:44 smithi142 ceph-mon[29958]: from='osd.1 v2:172.21.15.142:6805/599802221' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]': finished 2024-09-06T14:21:44.958 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:44 smithi142 ceph-mon[29958]: osdmap e12: 2 total, 1 up, 2 in 2024-09-06T14:21:44.958 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:44 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-06T14:21:44.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:44 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-06T14:21:45.046 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:44 smithi162 ceph-mon[33787]: from='osd.1 v2:172.21.15.142:6805/599802221' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]': finished 2024-09-06T14:21:45.046 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:44 smithi162 ceph-mon[33787]: osdmap e12: 2 total, 1 up, 2 in 2024-09-06T14:21:45.046 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:44 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-06T14:21:45.046 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:44 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-06T14:21:45.862 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:45 smithi142 ceph-mon[37372]: purged_snaps scrub starts 2024-09-06T14:21:45.862 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:45 smithi142 ceph-mon[37372]: purged_snaps scrub ok 2024-09-06T14:21:45.863 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:45 smithi142 ceph-mon[37372]: osd.1 v2:172.21.15.142:6805/599802221 boot 2024-09-06T14:21:45.863 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:45 smithi142 ceph-mon[37372]: osdmap e13: 2 total, 2 up, 2 in 2024-09-06T14:21:45.863 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:45 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-06T14:21:45.863 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:45 smithi142 ceph-mon[37372]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-06T14:21:45.863 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:45 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:21:45.863 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:45 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:21:45.863 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:45 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-06T14:21:45.863 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:45 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-06T14:21:45.863 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:45 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:21:45.863 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:45 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:21:45.863 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:45 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-06T14:21:45.863 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:45 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:21:45.864 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:45 smithi142 ceph-mon[29958]: purged_snaps scrub starts 2024-09-06T14:21:45.864 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:45 smithi142 ceph-mon[29958]: purged_snaps scrub ok 2024-09-06T14:21:45.864 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:45 smithi142 ceph-mon[29958]: osd.1 v2:172.21.15.142:6805/599802221 boot 2024-09-06T14:21:45.864 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:45 smithi142 ceph-mon[29958]: osdmap e13: 2 total, 2 up, 2 in 2024-09-06T14:21:45.864 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:45 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-06T14:21:45.864 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:45 smithi142 ceph-mon[29958]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-06T14:21:45.864 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:45 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:21:45.864 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:45 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:21:45.864 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:45 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-06T14:21:45.864 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:45 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-06T14:21:45.864 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:45 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:21:45.864 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:45 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:21:45.864 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:45 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-06T14:21:45.864 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:45 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:21:46.046 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:45 smithi162 ceph-mon[33787]: purged_snaps scrub starts 2024-09-06T14:21:46.046 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:45 smithi162 ceph-mon[33787]: purged_snaps scrub ok 2024-09-06T14:21:46.046 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:45 smithi162 ceph-mon[33787]: osd.1 v2:172.21.15.142:6805/599802221 boot 2024-09-06T14:21:46.046 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:45 smithi162 ceph-mon[33787]: osdmap e13: 2 total, 2 up, 2 in 2024-09-06T14:21:46.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:45 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-06T14:21:46.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:45 smithi162 ceph-mon[33787]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-06T14:21:46.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:45 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:21:46.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:45 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:21:46.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:45 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-06T14:21:46.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:45 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-06T14:21:46.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:45 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:21:46.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:45 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:21:46.048 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:45 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-06T14:21:46.048 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:45 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:21:46.758 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:46 smithi142 ceph-mon[29958]: Detected new or changed devices on smithi142 2024-09-06T14:21:46.758 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:46 smithi142 ceph-mon[29958]: Adjusting osd_memory_target on smithi142 to 8032M 2024-09-06T14:21:46.758 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:46 smithi142 ceph-mon[29958]: osdmap e14: 2 total, 2 up, 2 in 2024-09-06T14:21:46.759 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:46 smithi142 ceph-mon[37372]: Detected new or changed devices on smithi142 2024-09-06T14:21:46.759 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:46 smithi142 ceph-mon[37372]: Adjusting osd_memory_target on smithi142 to 8032M 2024-09-06T14:21:46.759 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:46 smithi142 ceph-mon[37372]: osdmap e14: 2 total, 2 up, 2 in 2024-09-06T14:21:46.761 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:21:46.788 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph orch daemon add osd smithi142:vg_nvme/lv_2 2024-09-06T14:21:47.035 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:21:47.046 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:46 smithi162 ceph-mon[33787]: Detected new or changed devices on smithi142 2024-09-06T14:21:47.046 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:46 smithi162 ceph-mon[33787]: Adjusting osd_memory_target on smithi142 to 8032M 2024-09-06T14:21:47.046 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:46 smithi162 ceph-mon[33787]: osdmap e14: 2 total, 2 up, 2 in 2024-09-06T14:21:47.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:47.523+0000 7fe104d71640 1 -- 172.21.15.142:0/3051361455 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe100104280 msgr2=0x7fe10010ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:21:47.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:47.523+0000 7fe104d71640 1 --2- 172.21.15.142:0/3051361455 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe100104280 0x7fe10010ac00 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fe0f8009cd0 tx=0x7fe0f802a8e0 comp rx=0 tx=0).stop 2024-09-06T14:21:47.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:47.523+0000 7fe104d71640 1 -- 172.21.15.142:0/3051361455 shutdown_connections 2024-09-06T14:21:47.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:47.523+0000 7fe104d71640 1 --2- 172.21.15.142:0/3051361455 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe100104280 0x7fe10010ac00 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:21:47.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:47.523+0000 7fe104d71640 1 --2- 172.21.15.142:0/3051361455 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe1001038c0 0x7fe100103d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:21:47.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:47.523+0000 7fe104d71640 1 --2- 172.21.15.142:0/3051361455 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe100102760 0x7fe100102b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:21:47.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:47.523+0000 7fe104d71640 1 -- 172.21.15.142:0/3051361455 >> 172.21.15.142:0/3051361455 conn(0x7fe1000fe050 msgr2=0x7fe100100470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:21:47.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:47.523+0000 7fe104d71640 1 -- 172.21.15.142:0/3051361455 shutdown_connections 2024-09-06T14:21:47.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:47.523+0000 7fe104d71640 1 -- 172.21.15.142:0/3051361455 wait complete. 2024-09-06T14:21:47.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:47.524+0000 7fe104d71640 1 Processor -- start 2024-09-06T14:21:47.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:47.524+0000 7fe104d71640 1 -- start start 2024-09-06T14:21:47.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:47.524+0000 7fe104d71640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe100102760 0x7fe1001981c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:21:47.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:47.524+0000 7fe104d71640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe1001038c0 0x7fe100198810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:21:47.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:47.524+0000 7fe104d71640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe100104280 0x7fe100199450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:21:47.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:47.524+0000 7fe104d71640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fe10010da70 con 0x7fe1001038c0 2024-09-06T14:21:47.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:47.524+0000 7fe104d71640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe10010d8f0 con 0x7fe100102760 2024-09-06T14:21:47.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:47.524+0000 7fe104d71640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7fe10010dbf0 con 0x7fe100104280 2024-09-06T14:21:47.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:47.524+0000 7fe0fdd9b640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe1001038c0 0x7fe100198810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:21:47.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:47.524+0000 7fe0fed9d640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe100104280 0x7fe100199450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:21:47.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:47.524+0000 7fe0fed9d640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe100104280 0x7fe100199450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:46968/0 (socket says 172.21.15.142:46968) 2024-09-06T14:21:47.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:47.524+0000 7fe0fdd9b640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe1001038c0 0x7fe100198810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:42386/0 (socket says 172.21.15.142:42386) 2024-09-06T14:21:47.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:47.524+0000 7fe0fed9d640 1 -- 172.21.15.142:0/58243717 learned_addr learned my addr 172.21.15.142:0/58243717 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:21:47.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:47.524+0000 7fe0fe59c640 1 --2- 172.21.15.142:0/58243717 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe100102760 0x7fe1001981c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:21:47.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:47.524+0000 7fe0fdd9b640 1 -- 172.21.15.142:0/58243717 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe100104280 msgr2=0x7fe100199450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:21:47.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:47.524+0000 7fe0fdd9b640 1 --2- 172.21.15.142:0/58243717 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe100104280 0x7fe100199450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:21:47.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:47.524+0000 7fe0fdd9b640 1 -- 172.21.15.142:0/58243717 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe100102760 msgr2=0x7fe1001981c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:21:47.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:47.524+0000 7fe0fdd9b640 1 --2- 172.21.15.142:0/58243717 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe100102760 0x7fe1001981c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:21:47.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:47.524+0000 7fe0fdd9b640 1 -- 172.21.15.142:0/58243717 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe100199cf0 con 0x7fe1001038c0 2024-09-06T14:21:47.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:47.524+0000 7fe0fe59c640 1 --2- 172.21.15.142:0/58243717 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe100102760 0x7fe1001981c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-06T14:21:47.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:47.524+0000 7fe0fdd9b640 1 --2- 172.21.15.142:0/58243717 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe1001038c0 0x7fe100198810 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fe0f400e9f0 tx=0x7fe0f400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:21:47.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:47.525+0000 7fe0ef7fe640 1 -- 172.21.15.142:0/58243717 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0f400cd80 con 0x7fe1001038c0 2024-09-06T14:21:47.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:47.525+0000 7fe0ef7fe640 1 -- 172.21.15.142:0/58243717 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7fe0f40035f0 con 0x7fe1001038c0 2024-09-06T14:21:47.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:47.525+0000 7fe104d71640 1 -- 172.21.15.142:0/58243717 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe100199fe0 con 0x7fe1001038c0 2024-09-06T14:21:47.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:47.525+0000 7fe0ef7fe640 1 -- 172.21.15.142:0/58243717 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0f40044a0 con 0x7fe1001038c0 2024-09-06T14:21:47.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:47.525+0000 7fe104d71640 1 -- 172.21.15.142:0/58243717 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1001a06c0 con 0x7fe1001038c0 2024-09-06T14:21:47.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:47.526+0000 7fe104d71640 1 -- 172.21.15.142:0/58243717 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe100102b60 con 0x7fe1001038c0 2024-09-06T14:21:47.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:47.526+0000 7fe0ef7fe640 1 -- 172.21.15.142:0/58243717 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 14) ==== 99730+0+0 (secure 0 0 0) 0x7fe0f4004000 con 0x7fe1001038c0 2024-09-06T14:21:47.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:47.527+0000 7fe0ef7fe640 1 --2- 172.21.15.142:0/58243717 >> v2:172.21.15.142:6800/853729507 conn(0x7fe0e80774a0 0x7fe0e8079960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:21:47.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:47.527+0000 7fe0fe59c640 1 --2- 172.21.15.142:0/58243717 >> v2:172.21.15.142:6800/853729507 conn(0x7fe0e80774a0 0x7fe0e8079960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:21:47.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:47.527+0000 7fe0ef7fe640 1 -- 172.21.15.142:0/58243717 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 1747+0+0 (secure 0 0 0) 0x7fe0f40988a0 con 0x7fe1001038c0 2024-09-06T14:21:47.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:47.527+0000 7fe0fe59c640 1 --2- 172.21.15.142:0/58243717 >> v2:172.21.15.142:6800/853729507 conn(0x7fe0e80774a0 0x7fe0e8079960 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe0f00058c0 tx=0x7fe0f0005d30 comp rx=0 tx=0).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:21:47.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:47.531+0000 7fe0ef7fe640 1 -- 172.21.15.142:0/58243717 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7fe0f4062c10 con 0x7fe1001038c0 2024-09-06T14:21:47.692 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:21:47.696+0000 7fe104d71640 1 -- 172.21.15.142:0/58243717 --> v2:172.21.15.142:6800/853729507 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi142:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fe1000632f0 con 0x7fe0e80774a0 2024-09-06T14:21:47.692 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:47 smithi142 ceph-mon[29958]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-06T14:21:47.693 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:47 smithi142 ceph-mon[37372]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-06T14:21:48.046 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:47 smithi162 ceph-mon[33787]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-06T14:21:48.708 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:48 smithi142 ceph-mon[29958]: from='client.14274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi142:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-06T14:21:48.708 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:48 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-06T14:21:48.708 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:48 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-06T14:21:48.708 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:48 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:21:48.708 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:48 smithi142 ceph-mon[37372]: from='client.14274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi142:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-06T14:21:48.708 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:48 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-06T14:21:48.709 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:48 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-06T14:21:48.709 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:48 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:21:49.046 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:48 smithi162 ceph-mon[33787]: from='client.14274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi142:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-06T14:21:49.046 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:48 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-06T14:21:49.046 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:48 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-06T14:21:49.046 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:48 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:21:49.707 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:49 smithi142 ceph-mon[29958]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-06T14:21:49.708 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:49 smithi142 ceph-mon[37372]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-06T14:21:50.048 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:49 smithi162 ceph-mon[33787]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-06T14:21:50.957 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:50 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/3838762616' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "d7b4c863-74aa-476b-9377-8262e2c77ae3"} : dispatch 2024-09-06T14:21:50.958 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:50 smithi142 ceph-mon[29958]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "d7b4c863-74aa-476b-9377-8262e2c77ae3"} : dispatch 2024-09-06T14:21:50.958 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:50 smithi142 ceph-mon[29958]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d7b4c863-74aa-476b-9377-8262e2c77ae3"}]': finished 2024-09-06T14:21:50.958 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:50 smithi142 ceph-mon[29958]: osdmap e15: 3 total, 2 up, 3 in 2024-09-06T14:21:50.958 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:50 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-06T14:21:50.958 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:50 smithi142 ceph-mon[37372]: from='client.? 172.21.15.142:0/3838762616' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "d7b4c863-74aa-476b-9377-8262e2c77ae3"} : dispatch 2024-09-06T14:21:50.958 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:50 smithi142 ceph-mon[37372]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "d7b4c863-74aa-476b-9377-8262e2c77ae3"} : dispatch 2024-09-06T14:21:50.958 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:50 smithi142 ceph-mon[37372]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d7b4c863-74aa-476b-9377-8262e2c77ae3"}]': finished 2024-09-06T14:21:50.958 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:50 smithi142 ceph-mon[37372]: osdmap e15: 3 total, 2 up, 3 in 2024-09-06T14:21:50.958 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:50 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-06T14:21:51.046 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:50 smithi162 ceph-mon[33787]: from='client.? 172.21.15.142:0/3838762616' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "d7b4c863-74aa-476b-9377-8262e2c77ae3"} : dispatch 2024-09-06T14:21:51.046 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:50 smithi162 ceph-mon[33787]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "d7b4c863-74aa-476b-9377-8262e2c77ae3"} : dispatch 2024-09-06T14:21:51.046 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:50 smithi162 ceph-mon[33787]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d7b4c863-74aa-476b-9377-8262e2c77ae3"}]': finished 2024-09-06T14:21:51.046 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:50 smithi162 ceph-mon[33787]: osdmap e15: 3 total, 2 up, 3 in 2024-09-06T14:21:51.046 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:50 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-06T14:21:51.957 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:51 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/3345400346' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-06T14:21:51.958 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:51 smithi142 ceph-mon[29958]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-06T14:21:51.958 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:51 smithi142 ceph-mon[37372]: from='client.? 172.21.15.142:0/3345400346' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-06T14:21:51.958 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:51 smithi142 ceph-mon[37372]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-06T14:21:52.046 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:51 smithi162 ceph-mon[33787]: from='client.? 172.21.15.142:0/3345400346' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-06T14:21:52.046 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:51 smithi162 ceph-mon[33787]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-06T14:21:54.207 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:53 smithi142 ceph-mon[29958]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-06T14:21:54.208 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:53 smithi142 ceph-mon[37372]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-06T14:21:54.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:53 smithi162 ceph-mon[33787]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-06T14:21:56.049 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:55 smithi142 ceph-mon[29958]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-06T14:21:56.049 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:55 smithi142 ceph-mon[37372]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-06T14:21:56.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:55 smithi162 ceph-mon[33787]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-06T14:21:58.060 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:57 smithi142 ceph-mon[29958]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-06T14:21:58.061 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:57 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-06T14:21:58.061 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:57 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:21:58.061 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:57 smithi142 ceph-mon[37372]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-06T14:21:58.061 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:57 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-06T14:21:58.061 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:57 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:21:58.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:57 smithi162 ceph-mon[33787]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-06T14:21:58.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:57 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-06T14:21:58.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:57 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:21:59.058 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:58 smithi142 ceph-mon[29958]: Deploying daemon osd.2 on smithi142 2024-09-06T14:21:59.058 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:58 smithi142 ceph-mon[37372]: Deploying daemon osd.2 on smithi142 2024-09-06T14:21:59.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:58 smithi162 ceph-mon[33787]: Deploying daemon osd.2 on smithi142 2024-09-06T14:22:00.192 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:21:59 smithi142 ceph-mon[29958]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-06T14:22:00.192 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:21:59 smithi142 ceph-mon[37372]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-06T14:22:00.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:21:59 smithi162 ceph-mon[33787]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-06T14:22:02.189 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:01 smithi142 ceph-mon[29958]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-06T14:22:02.189 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:01 smithi142 ceph-mon[37372]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-06T14:22:02.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:01 smithi162 ceph-mon[33787]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-06T14:22:04.133 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:03 smithi142 ceph-mon[37372]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-06T14:22:04.133 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:03 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-06T14:22:04.133 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:03 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:22:04.134 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:03 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:22:04.134 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:03 smithi142 ceph-mon[29958]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-06T14:22:04.134 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:03 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-06T14:22:04.134 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:03 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:22:04.134 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:03 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:22:04.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:03 smithi162 ceph-mon[33787]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-06T14:22:04.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:03 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-06T14:22:04.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:03 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:22:04.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:03 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:22:05.951 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:05 smithi142 ceph-mon[29958]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-06T14:22:05.952 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:05 smithi142 ceph-mon[37372]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-06T14:22:06.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:05 smithi162 ceph-mon[33787]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-06T14:22:07.221 INFO:teuthology.orchestra.run.smithi142.stdout:Created osd(s) 2 on host 'smithi142' 2024-09-06T14:22:07.222 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:22:07.225+0000 7fe0ef7fe640 1 -- 172.21.15.142:0/58243717 <== mgr.14146 v2:172.21.15.142:6800/853729507 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fe1000632f0 con 0x7fe0e80774a0 2024-09-06T14:22:07.224 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:22:07.228+0000 7fe104d71640 1 -- 172.21.15.142:0/58243717 >> v2:172.21.15.142:6800/853729507 conn(0x7fe0e80774a0 msgr2=0x7fe0e8079960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:22:07.224 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:22:07.228+0000 7fe104d71640 1 --2- 172.21.15.142:0/58243717 >> v2:172.21.15.142:6800/853729507 conn(0x7fe0e80774a0 0x7fe0e8079960 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe0f00058c0 tx=0x7fe0f0005d30 comp rx=0 tx=0).stop 2024-09-06T14:22:07.224 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:22:07.228+0000 7fe104d71640 1 -- 172.21.15.142:0/58243717 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe1001038c0 msgr2=0x7fe100198810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:22:07.224 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:22:07.228+0000 7fe104d71640 1 --2- 172.21.15.142:0/58243717 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe1001038c0 0x7fe100198810 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fe0f400e9f0 tx=0x7fe0f400eeb0 comp rx=0 tx=0).stop 2024-09-06T14:22:07.224 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:22:07.228+0000 7fe104d71640 1 -- 172.21.15.142:0/58243717 shutdown_connections 2024-09-06T14:22:07.224 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:22:07.228+0000 7fe104d71640 1 --2- 172.21.15.142:0/58243717 >> v2:172.21.15.142:6800/853729507 conn(0x7fe0e80774a0 0x7fe0e8079960 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:22:07.224 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:22:07.228+0000 7fe104d71640 1 --2- 172.21.15.142:0/58243717 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe100104280 0x7fe100199450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:22:07.224 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:22:07.228+0000 7fe104d71640 1 --2- 172.21.15.142:0/58243717 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe1001038c0 0x7fe100198810 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:22:07.224 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:22:07.228+0000 7fe104d71640 1 --2- 172.21.15.142:0/58243717 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe100102760 0x7fe1001981c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:22:07.225 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:22:07.228+0000 7fe104d71640 1 -- 172.21.15.142:0/58243717 >> 172.21.15.142:0/58243717 conn(0x7fe1000fe050 msgr2=0x7fe1000ffae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:22:07.225 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:22:07.228+0000 7fe104d71640 1 -- 172.21.15.142:0/58243717 shutdown_connections 2024-09-06T14:22:07.225 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:22:07.228+0000 7fe104d71640 1 -- 172.21.15.142:0/58243717 wait complete. 2024-09-06T14:22:07.461 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:07 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:22:07.461 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:07 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:22:07.461 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:07 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:22:07.461 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:07 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-06T14:22:07.461 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:07 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:22:07.461 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:07 smithi142 ceph-mon[29958]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-06T14:22:07.461 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:07 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-06T14:22:07.461 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:07 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:22:07.461 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:07 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:22:07.462 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:07 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:22:07.462 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:07 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:22:07.462 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:07 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:22:07.462 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:07 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-06T14:22:07.462 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:07 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:22:07.462 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:07 smithi142 ceph-mon[37372]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-06T14:22:07.462 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:07 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-06T14:22:07.462 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:07 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:22:07.462 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:07 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:22:07.546 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:07 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:22:07.546 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:07 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:22:07.546 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:07 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:22:07.546 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:07 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-06T14:22:07.547 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:07 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:22:07.547 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:07 smithi162 ceph-mon[33787]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-06T14:22:07.547 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:07 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-06T14:22:07.547 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:07 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:22:07.547 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:07 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:22:07.878 DEBUG:teuthology.orchestra.run.smithi142:osd.2> sudo journalctl -f -n 0 -u ceph-ce32462e-6c5a-11ef-bce4-c7b262605968@osd.2.service 2024-09-06T14:22:07.881 INFO:tasks.cephadm:Deploying osd.3 on smithi142 with /dev/vg_nvme/lv_1... 2024-09-06T14:22:07.881 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- lvm zap /dev/vg_nvme/lv_1 2024-09-06T14:22:08.458 INFO:journalctl@ceph.osd.2.smithi142.stdout:Sep 06 14:22:08 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-2[52762]: 2024-09-06T14:22:08.450+0000 7f2d04279640 -1 osd.2 0 log_to_monitors true 2024-09-06T14:22:08.796 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:08 smithi162 ceph-mon[33787]: from='osd.2 v2:172.21.15.142:6809/1302270091' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-06T14:22:08.796 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:08 smithi162 ceph-mon[33787]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-06T14:22:08.845 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:08 smithi142 ceph-mon[29958]: from='osd.2 v2:172.21.15.142:6809/1302270091' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-06T14:22:08.845 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:08 smithi142 ceph-mon[29958]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-06T14:22:08.845 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:08 smithi142 ceph-mon[37372]: from='osd.2 v2:172.21.15.142:6809/1302270091' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-06T14:22:08.845 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:08 smithi142 ceph-mon[37372]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-06T14:22:09.315 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:22:09.929 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:09 smithi142 ceph-mon[29958]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-06T14:22:09.929 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:09 smithi142 ceph-mon[29958]: from='osd.2 v2:172.21.15.142:6809/1302270091' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi142", "root=default"]} : dispatch 2024-09-06T14:22:09.929 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:09 smithi142 ceph-mon[29958]: osdmap e16: 3 total, 2 up, 3 in 2024-09-06T14:22:09.930 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:09 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-06T14:22:09.930 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:09 smithi142 ceph-mon[29958]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi142", "root=default"]} : dispatch 2024-09-06T14:22:09.930 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:09 smithi142 ceph-mon[29958]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-06T14:22:09.930 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:09 smithi142 ceph-mon[29958]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]': finished 2024-09-06T14:22:09.930 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:09 smithi142 ceph-mon[29958]: osdmap e17: 3 total, 2 up, 3 in 2024-09-06T14:22:09.930 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:09 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-06T14:22:09.930 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:09 smithi142 ceph-mon[37372]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-06T14:22:09.930 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:09 smithi142 ceph-mon[37372]: from='osd.2 v2:172.21.15.142:6809/1302270091' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi142", "root=default"]} : dispatch 2024-09-06T14:22:09.930 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:09 smithi142 ceph-mon[37372]: osdmap e16: 3 total, 2 up, 3 in 2024-09-06T14:22:09.931 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:09 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-06T14:22:09.931 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:09 smithi142 ceph-mon[37372]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi142", "root=default"]} : dispatch 2024-09-06T14:22:09.931 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:09 smithi142 ceph-mon[37372]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-06T14:22:09.931 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:09 smithi142 ceph-mon[37372]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]': finished 2024-09-06T14:22:09.931 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:09 smithi142 ceph-mon[37372]: osdmap e17: 3 total, 2 up, 3 in 2024-09-06T14:22:09.931 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:09 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-06T14:22:10.046 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:09 smithi162 ceph-mon[33787]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-06T14:22:10.046 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:09 smithi162 ceph-mon[33787]: from='osd.2 v2:172.21.15.142:6809/1302270091' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi142", "root=default"]} : dispatch 2024-09-06T14:22:10.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:09 smithi162 ceph-mon[33787]: osdmap e16: 3 total, 2 up, 3 in 2024-09-06T14:22:10.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:09 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-06T14:22:10.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:09 smithi162 ceph-mon[33787]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi142", "root=default"]} : dispatch 2024-09-06T14:22:10.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:09 smithi162 ceph-mon[33787]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-06T14:22:10.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:09 smithi162 ceph-mon[33787]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]': finished 2024-09-06T14:22:10.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:09 smithi162 ceph-mon[33787]: osdmap e17: 3 total, 2 up, 3 in 2024-09-06T14:22:10.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:09 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-06T14:22:10.208 INFO:journalctl@ceph.osd.2.smithi142.stdout:Sep 06 14:22:09 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-2[52762]: 2024-09-06T14:22:09.952+0000 7f2d001fc640 -1 osd.2 0 waiting for initial osdmap 2024-09-06T14:22:10.208 INFO:journalctl@ceph.osd.2.smithi142.stdout:Sep 06 14:22:09 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-2[52762]: 2024-09-06T14:22:09.955+0000 7f2cf801b640 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-06T14:22:10.935 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:10 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-06T14:22:10.935 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:10 smithi142 ceph-mon[29958]: from='osd.2 ' entity='osd.2' 2024-09-06T14:22:10.935 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:10 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-06T14:22:10.935 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:10 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-06T14:22:10.936 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:10 smithi142 ceph-mon[37372]: from='osd.2 ' entity='osd.2' 2024-09-06T14:22:10.936 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:10 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-06T14:22:11.046 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:10 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-06T14:22:11.046 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:10 smithi162 ceph-mon[33787]: from='osd.2 ' entity='osd.2' 2024-09-06T14:22:11.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:10 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-06T14:22:11.708 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:11 smithi142 ceph-mon[37372]: purged_snaps scrub starts 2024-09-06T14:22:11.708 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:11 smithi142 ceph-mon[37372]: purged_snaps scrub ok 2024-09-06T14:22:11.708 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:11 smithi142 ceph-mon[37372]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-06T14:22:11.708 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:11 smithi142 ceph-mon[37372]: osd.2 v2:172.21.15.142:6809/1302270091 boot 2024-09-06T14:22:11.708 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:11 smithi142 ceph-mon[37372]: osdmap e18: 3 total, 3 up, 3 in 2024-09-06T14:22:11.708 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:11 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-06T14:22:11.708 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:11 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true} : dispatch 2024-09-06T14:22:11.708 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:11 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:22:11.709 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:11 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:22:11.709 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:11 smithi142 ceph-mon[29958]: purged_snaps scrub starts 2024-09-06T14:22:11.709 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:11 smithi142 ceph-mon[29958]: purged_snaps scrub ok 2024-09-06T14:22:11.709 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:11 smithi142 ceph-mon[29958]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-06T14:22:11.709 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:11 smithi142 ceph-mon[29958]: osd.2 v2:172.21.15.142:6809/1302270091 boot 2024-09-06T14:22:11.709 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:11 smithi142 ceph-mon[29958]: osdmap e18: 3 total, 3 up, 3 in 2024-09-06T14:22:11.709 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:11 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-06T14:22:11.709 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:11 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true} : dispatch 2024-09-06T14:22:11.709 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:11 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:22:11.709 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:11 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:22:12.046 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:11 smithi162 ceph-mon[33787]: purged_snaps scrub starts 2024-09-06T14:22:12.046 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:11 smithi162 ceph-mon[33787]: purged_snaps scrub ok 2024-09-06T14:22:12.046 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:11 smithi162 ceph-mon[33787]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-06T14:22:12.046 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:11 smithi162 ceph-mon[33787]: osd.2 v2:172.21.15.142:6809/1302270091 boot 2024-09-06T14:22:12.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:11 smithi162 ceph-mon[33787]: osdmap e18: 3 total, 3 up, 3 in 2024-09-06T14:22:12.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:11 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-06T14:22:12.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:11 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true} : dispatch 2024-09-06T14:22:12.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:11 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:22:12.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:11 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:22:12.708 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:12 smithi142 ceph-mon[29958]: Detected new or changed devices on smithi142 2024-09-06T14:22:12.708 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:12 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-06T14:22:12.708 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:12 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-06T14:22:12.708 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:12 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-06T14:22:12.708 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:12 smithi142 ceph-mon[29958]: Adjusting osd_memory_target on smithi142 to 5354M 2024-09-06T14:22:12.708 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:12 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:22:12.708 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:12 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:22:12.709 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:12 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-06T14:22:12.709 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:12 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:22:12.709 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:12 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-06T14:22:12.709 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:12 smithi142 ceph-mon[29958]: osdmap e19: 3 total, 3 up, 3 in 2024-09-06T14:22:12.709 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:12 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true} : dispatch 2024-09-06T14:22:12.709 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:12 smithi142 ceph-mon[37372]: Detected new or changed devices on smithi142 2024-09-06T14:22:12.709 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:12 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-06T14:22:12.709 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:12 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-06T14:22:12.709 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:12 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-06T14:22:12.709 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:12 smithi142 ceph-mon[37372]: Adjusting osd_memory_target on smithi142 to 5354M 2024-09-06T14:22:12.709 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:12 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:22:12.709 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:12 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:22:12.710 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:12 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-06T14:22:12.710 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:12 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:22:12.710 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:12 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-06T14:22:12.710 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:12 smithi142 ceph-mon[37372]: osdmap e19: 3 total, 3 up, 3 in 2024-09-06T14:22:12.710 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:12 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true} : dispatch 2024-09-06T14:22:12.947 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:12 smithi162 ceph-mon[33787]: Detected new or changed devices on smithi142 2024-09-06T14:22:12.948 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:12 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-06T14:22:12.948 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:12 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-06T14:22:12.948 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:12 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-06T14:22:12.948 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:12 smithi162 ceph-mon[33787]: Adjusting osd_memory_target on smithi142 to 5354M 2024-09-06T14:22:12.948 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:12 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:22:12.948 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:12 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:22:12.948 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:12 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-06T14:22:12.948 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:12 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:22:12.948 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:12 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-06T14:22:12.948 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:12 smithi162 ceph-mon[33787]: osdmap e19: 3 total, 3 up, 3 in 2024-09-06T14:22:12.948 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:12 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true} : dispatch 2024-09-06T14:22:13.063 INFO:journalctl@ceph.osd.0.smithi142.stdout:Sep 06 14:22:13 smithi142 sudo[56409]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-06T14:22:13.064 INFO:journalctl@ceph.osd.0.smithi142.stdout:Sep 06 14:22:13 smithi142 sudo[56409]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-06T14:22:13.064 INFO:journalctl@ceph.osd.0.smithi142.stdout:Sep 06 14:22:13 smithi142 sudo[56409]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-06T14:22:13.064 INFO:journalctl@ceph.osd.0.smithi142.stdout:Sep 06 14:22:13 smithi142 sudo[56409]: pam_unix(sudo:session): session closed for user root 2024-09-06T14:22:13.064 INFO:journalctl@ceph.osd.0.smithi142.stdout:Sep 06 14:22:13 smithi142 sudo[56412]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-06T14:22:13.064 INFO:journalctl@ceph.osd.0.smithi142.stdout:Sep 06 14:22:13 smithi142 sudo[56412]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-06T14:22:13.064 INFO:journalctl@ceph.osd.0.smithi142.stdout:Sep 06 14:22:13 smithi142 sudo[56412]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-06T14:22:13.064 INFO:journalctl@ceph.osd.0.smithi142.stdout:Sep 06 14:22:13 smithi142 sudo[56412]: pam_unix(sudo:session): session closed for user root 2024-09-06T14:22:13.110 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:22:13.139 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph orch daemon add osd smithi142:vg_nvme/lv_1 2024-09-06T14:22:13.204 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:13 smithi162 sudo[36250]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-06T14:22:13.205 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:13 smithi162 sudo[36250]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-06T14:22:13.205 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:13 smithi162 sudo[36250]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-06T14:22:13.205 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:13 smithi162 sudo[36250]: pam_unix(sudo:session): session closed for user root 2024-09-06T14:22:13.400 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:22:13.457 INFO:journalctl@ceph.osd.2.smithi142.stdout:Sep 06 14:22:13 smithi142 sudo[56421]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-06T14:22:13.458 INFO:journalctl@ceph.osd.2.smithi142.stdout:Sep 06 14:22:13 smithi142 sudo[56421]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-06T14:22:13.458 INFO:journalctl@ceph.osd.2.smithi142.stdout:Sep 06 14:22:13 smithi142 sudo[56421]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-06T14:22:13.458 INFO:journalctl@ceph.osd.2.smithi142.stdout:Sep 06 14:22:13 smithi142 sudo[56421]: pam_unix(sudo:session): session closed for user root 2024-09-06T14:22:13.458 INFO:journalctl@ceph.osd.2.smithi142.stdout:Sep 06 14:22:13 smithi142 sudo[56424]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-06T14:22:13.458 INFO:journalctl@ceph.osd.2.smithi142.stdout:Sep 06 14:22:13 smithi142 sudo[56424]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-06T14:22:13.458 INFO:journalctl@ceph.osd.2.smithi142.stdout:Sep 06 14:22:13 smithi142 sudo[56424]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-06T14:22:13.458 INFO:journalctl@ceph.osd.2.smithi142.stdout:Sep 06 14:22:13 smithi142 sudo[56424]: pam_unix(sudo:session): session closed for user root 2024-09-06T14:22:13.458 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:13 smithi142 sudo[56427]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-06T14:22:13.459 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:13 smithi142 sudo[56427]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-06T14:22:13.459 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:13 smithi142 sudo[56427]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-06T14:22:13.459 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:13 smithi142 sudo[56427]: pam_unix(sudo:session): session closed for user root 2024-09-06T14:22:13.459 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:13 smithi142 sudo[56447]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-06T14:22:13.459 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:13 smithi142 sudo[56447]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-06T14:22:13.459 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:13 smithi142 sudo[56447]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-06T14:22:13.459 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:13 smithi142 sudo[56447]: pam_unix(sudo:session): session closed for user root 2024-09-06T14:22:13.459 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:13 smithi142 sudo[56451]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-06T14:22:13.459 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:13 smithi142 sudo[56451]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-06T14:22:13.459 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:13 smithi142 sudo[56451]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-06T14:22:13.459 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:13 smithi142 sudo[56451]: pam_unix(sudo:session): session closed for user root 2024-09-06T14:22:13.460 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:13 smithi142 sudo[56454]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-06T14:22:13.460 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:13 smithi142 sudo[56454]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-06T14:22:13.460 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:13 smithi142 sudo[56454]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-06T14:22:13.460 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:13 smithi142 sudo[56454]: pam_unix(sudo:session): session closed for user root 2024-09-06T14:22:13.460 INFO:journalctl@ceph.osd.1.smithi142.stdout:Sep 06 14:22:13 smithi142 sudo[56415]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-06T14:22:13.460 INFO:journalctl@ceph.osd.1.smithi142.stdout:Sep 06 14:22:13 smithi142 sudo[56415]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-06T14:22:13.460 INFO:journalctl@ceph.osd.1.smithi142.stdout:Sep 06 14:22:13 smithi142 sudo[56415]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-06T14:22:13.460 INFO:journalctl@ceph.osd.1.smithi142.stdout:Sep 06 14:22:13 smithi142 sudo[56415]: pam_unix(sudo:session): session closed for user root 2024-09-06T14:22:13.460 INFO:journalctl@ceph.osd.1.smithi142.stdout:Sep 06 14:22:13 smithi142 sudo[56418]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-06T14:22:13.460 INFO:journalctl@ceph.osd.1.smithi142.stdout:Sep 06 14:22:13 smithi142 sudo[56418]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-06T14:22:13.460 INFO:journalctl@ceph.osd.1.smithi142.stdout:Sep 06 14:22:13 smithi142 sudo[56418]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-06T14:22:13.461 INFO:journalctl@ceph.osd.1.smithi142.stdout:Sep 06 14:22:13 smithi142 sudo[56418]: pam_unix(sudo:session): session closed for user root 2024-09-06T14:22:13.546 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:13 smithi162 sudo[36253]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-06T14:22:13.546 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:13 smithi162 sudo[36253]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-06T14:22:13.546 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:13 smithi162 sudo[36253]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-06T14:22:13.546 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:13 smithi162 sudo[36253]: pam_unix(sudo:session): session closed for user root 2024-09-06T14:22:13.809 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:13 smithi142 ceph-mon[37372]: pgmap v76: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-06T14:22:13.809 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:13 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-06T14:22:13.809 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:13 smithi142 ceph-mon[37372]: osdmap e20: 3 total, 3 up, 3 in 2024-09-06T14:22:13.809 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:13 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:22:13.809 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:13 smithi142 ceph-mon[37372]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-06T14:22:13.809 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:13 smithi142 ceph-mon[37372]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-06T14:22:13.809 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:13 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-06T14:22:13.809 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:13 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-06T14:22:13.809 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:13 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-06T14:22:13.810 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:13 smithi142 ceph-mon[37372]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-06T14:22:13.810 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:13 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-06T14:22:13.810 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:13 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-06T14:22:13.810 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:13 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-06T14:22:13.810 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:13 smithi142 ceph-mon[37372]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-06T14:22:13.810 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:13 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:22:13.810 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:13 smithi142 ceph-mon[37372]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-06T14:22:13.810 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:13 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-06T14:22:13.810 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:13 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-06T14:22:13.810 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:13 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-06T14:22:13.810 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:13 smithi142 ceph-mon[37372]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-06T14:22:13.810 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:13 smithi142 ceph-mon[29958]: pgmap v76: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-06T14:22:13.811 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:13 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-06T14:22:13.811 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:13 smithi142 ceph-mon[29958]: osdmap e20: 3 total, 3 up, 3 in 2024-09-06T14:22:13.811 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:13 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:22:13.811 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:13 smithi142 ceph-mon[29958]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-06T14:22:13.811 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:13 smithi142 ceph-mon[29958]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-06T14:22:13.811 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:13 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-06T14:22:13.811 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:13 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-06T14:22:13.811 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:13 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-06T14:22:13.811 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:13 smithi142 ceph-mon[29958]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-06T14:22:13.811 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:13 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-06T14:22:13.811 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:13 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-06T14:22:13.811 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:13 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-06T14:22:13.811 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:13 smithi142 ceph-mon[29958]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-06T14:22:13.812 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:13 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:22:13.812 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:13 smithi142 ceph-mon[29958]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-06T14:22:13.812 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:13 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-06T14:22:13.812 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:13 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-06T14:22:13.812 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:13 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-06T14:22:13.812 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:13 smithi142 ceph-mon[29958]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-06T14:22:13.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:22:13.897+0000 7f9811ba5640 1 -- 172.21.15.142:0/1182409276 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f980c104280 msgr2=0x7f980c10ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:22:13.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:22:13.897+0000 7f9811ba5640 1 --2- 172.21.15.142:0/1182409276 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f980c104280 0x7f980c10ac00 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f9804009d00 tx=0x7f980402a910 comp rx=0 tx=0).stop 2024-09-06T14:22:13.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:22:13.897+0000 7f9811ba5640 1 -- 172.21.15.142:0/1182409276 shutdown_connections 2024-09-06T14:22:13.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:22:13.897+0000 7f9811ba5640 1 --2- 172.21.15.142:0/1182409276 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f980c104280 0x7f980c10ac00 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:22:13.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:22:13.897+0000 7f9811ba5640 1 --2- 172.21.15.142:0/1182409276 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f980c1038c0 0x7f980c103d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:22:13.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:22:13.897+0000 7f9811ba5640 1 --2- 172.21.15.142:0/1182409276 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f980c102760 0x7f980c102b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:22:13.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:22:13.897+0000 7f9811ba5640 1 -- 172.21.15.142:0/1182409276 >> 172.21.15.142:0/1182409276 conn(0x7f980c0fe050 msgr2=0x7f980c100470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:22:13.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:22:13.897+0000 7f9811ba5640 1 -- 172.21.15.142:0/1182409276 shutdown_connections 2024-09-06T14:22:13.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:22:13.897+0000 7f9811ba5640 1 -- 172.21.15.142:0/1182409276 wait complete. 2024-09-06T14:22:13.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:22:13.898+0000 7f9811ba5640 1 Processor -- start 2024-09-06T14:22:13.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:22:13.898+0000 7f9811ba5640 1 -- start start 2024-09-06T14:22:13.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:22:13.898+0000 7f9811ba5640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f980c102760 0x7f980c198170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:22:13.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:22:13.898+0000 7f9811ba5640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f980c1038c0 0x7f980c1987c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:22:13.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:22:13.898+0000 7f9811ba5640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f980c104280 0x7f980c199400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:22:13.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:22:13.898+0000 7f9811ba5640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f980c10dac0 con 0x7f980c102760 2024-09-06T14:22:13.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:22:13.899+0000 7f9811ba5640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f980c10d940 con 0x7f980c1038c0 2024-09-06T14:22:13.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:22:13.899+0000 7f9811ba5640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7f980c10dc40 con 0x7f980c104280 2024-09-06T14:22:13.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:22:13.899+0000 7f980b7fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f980c102760 0x7f980c198170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:22:13.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:22:13.899+0000 7f980b7fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f980c102760 0x7f980c198170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:40270/0 (socket says 172.21.15.142:40270) 2024-09-06T14:22:13.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:22:13.899+0000 7f980b7fe640 1 -- 172.21.15.142:0/3059614526 learned_addr learned my addr 172.21.15.142:0/3059614526 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:22:13.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:22:13.899+0000 7f980bfff640 1 --2- 172.21.15.142:0/3059614526 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f980c104280 0x7f980c199400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:22:13.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:22:13.899+0000 7f980affd640 1 --2- 172.21.15.142:0/3059614526 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f980c1038c0 0x7f980c1987c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:22:13.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:22:13.899+0000 7f980b7fe640 1 -- 172.21.15.142:0/3059614526 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f980c104280 msgr2=0x7f980c199400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:22:13.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:22:13.899+0000 7f980b7fe640 1 --2- 172.21.15.142:0/3059614526 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f980c104280 0x7f980c199400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:22:13.913 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:22:13.899+0000 7f980b7fe640 1 -- 172.21.15.142:0/3059614526 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f980c1038c0 msgr2=0x7f980c1987c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:22:13.913 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:22:13.899+0000 7f980b7fe640 1 --2- 172.21.15.142:0/3059614526 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f980c1038c0 0x7f980c1987c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:22:13.913 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:22:13.899+0000 7f980b7fe640 1 -- 172.21.15.142:0/3059614526 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f980c199ca0 con 0x7f980c102760 2024-09-06T14:22:13.913 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:22:13.899+0000 7f980bfff640 1 --2- 172.21.15.142:0/3059614526 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f980c104280 0x7f980c199400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-06T14:22:13.913 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:22:13.899+0000 7f980b7fe640 1 --2- 172.21.15.142:0/3059614526 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f980c102760 0x7f980c198170 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f97fc00c920 tx=0x7f97fc00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:22:13.913 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:22:13.899+0000 7f9808ff9640 1 -- 172.21.15.142:0/3059614526 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97fc007c70 con 0x7f980c102760 2024-09-06T14:22:13.913 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:22:13.899+0000 7f9808ff9640 1 -- 172.21.15.142:0/3059614526 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7f97fc007e10 con 0x7f980c102760 2024-09-06T14:22:13.913 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:22:13.899+0000 7f9811ba5640 1 -- 172.21.15.142:0/3059614526 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f980c199f90 con 0x7f980c102760 2024-09-06T14:22:13.913 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:22:13.899+0000 7f9808ff9640 1 -- 172.21.15.142:0/3059614526 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97fc018700 con 0x7f980c102760 2024-09-06T14:22:13.913 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:22:13.899+0000 7f9811ba5640 1 -- 172.21.15.142:0/3059614526 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f980c1a0670 con 0x7f980c102760 2024-09-06T14:22:13.913 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:22:13.900+0000 7f9811ba5640 1 -- 172.21.15.142:0/3059614526 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f980c102b60 con 0x7f980c102760 2024-09-06T14:22:13.913 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:22:13.901+0000 7f9808ff9640 1 -- 172.21.15.142:0/3059614526 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 14) ==== 99730+0+0 (secure 0 0 0) 0x7f97fc00f450 con 0x7f980c102760 2024-09-06T14:22:13.913 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:22:13.901+0000 7f9808ff9640 1 --2- 172.21.15.142:0/3059614526 >> v2:172.21.15.142:6800/853729507 conn(0x7f97f40774a0 0x7f97f4079960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:22:13.913 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:22:13.902+0000 7f9808ff9640 1 -- 172.21.15.142:0/3059614526 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 2425+0+0 (secure 0 0 0) 0x7f97fc0988d0 con 0x7f980c102760 2024-09-06T14:22:13.914 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:22:13.902+0000 7f980affd640 1 --2- 172.21.15.142:0/3059614526 >> v2:172.21.15.142:6800/853729507 conn(0x7f97f40774a0 0x7f97f4079960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:22:13.914 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:22:13.902+0000 7f980affd640 1 --2- 172.21.15.142:0/3059614526 >> v2:172.21.15.142:6800/853729507 conn(0x7f97f40774a0 0x7f97f4079960 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f9800009bd0 tx=0x7f9800005f90 comp rx=0 tx=0).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:22:13.914 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:22:13.906+0000 7f9808ff9640 1 -- 172.21.15.142:0/3059614526 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7f97fc014030 con 0x7f980c102760 2024-09-06T14:22:14.046 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:13 smithi162 ceph-mon[33787]: pgmap v76: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-06T14:22:14.046 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:13 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-06T14:22:14.046 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:13 smithi162 ceph-mon[33787]: osdmap e20: 3 total, 3 up, 3 in 2024-09-06T14:22:14.046 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:13 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:22:14.046 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:13 smithi162 ceph-mon[33787]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-06T14:22:14.046 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:13 smithi162 ceph-mon[33787]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-06T14:22:14.046 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:13 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-06T14:22:14.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:13 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-06T14:22:14.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:13 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-06T14:22:14.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:13 smithi162 ceph-mon[33787]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-06T14:22:14.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:13 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-06T14:22:14.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:13 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-06T14:22:14.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:13 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-06T14:22:14.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:13 smithi162 ceph-mon[33787]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-06T14:22:14.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:13 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:22:14.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:13 smithi162 ceph-mon[33787]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-06T14:22:14.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:13 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-06T14:22:14.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:13 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-06T14:22:14.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:13 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-06T14:22:14.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:13 smithi162 ceph-mon[33787]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-06T14:22:14.063 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:22:14.067+0000 7f9811ba5640 1 -- 172.21.15.142:0/3059614526 --> v2:172.21.15.142:6800/853729507 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi142:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f980c0632f0 con 0x7f97f40774a0 2024-09-06T14:22:14.720 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:14 smithi142 ceph-mon[29958]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-06T14:22:14.721 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:14 smithi142 ceph-mon[29958]: osdmap e21: 3 total, 3 up, 3 in 2024-09-06T14:22:14.721 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:14 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-06T14:22:14.721 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:14 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-06T14:22:14.721 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:14 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:22:14.721 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:14 smithi142 ceph-mon[37372]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-06T14:22:14.721 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:14 smithi142 ceph-mon[37372]: osdmap e21: 3 total, 3 up, 3 in 2024-09-06T14:22:14.721 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:14 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-06T14:22:14.721 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:14 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-06T14:22:14.721 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:14 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:22:15.046 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:14 smithi162 ceph-mon[33787]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-06T14:22:15.046 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:14 smithi162 ceph-mon[33787]: osdmap e21: 3 total, 3 up, 3 in 2024-09-06T14:22:15.046 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:14 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-06T14:22:15.046 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:14 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-06T14:22:15.046 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:14 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:22:15.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:22:15.655+0000 7f9808ff9640 1 -- 172.21.15.142:0/3059614526 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mgrmap(e 15) ==== 99786+0+0 (secure 0 0 0) 0x7f97fc05e540 con 0x7f980c102760 2024-09-06T14:22:15.818 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:15 smithi142 ceph-mon[37372]: from='client.14310 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi142:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-06T14:22:15.818 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:15 smithi142 ceph-mon[37372]: pgmap v79: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-06T14:22:15.818 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:15 smithi142 ceph-mon[29958]: from='client.14310 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi142:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-06T14:22:15.818 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:15 smithi142 ceph-mon[29958]: pgmap v79: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-06T14:22:16.046 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:15 smithi162 ceph-mon[33787]: from='client.14310 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi142:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-06T14:22:16.046 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:15 smithi162 ceph-mon[33787]: pgmap v79: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-06T14:22:16.958 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:16 smithi142 ceph-mon[37372]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-06T14:22:16.958 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:16 smithi142 ceph-mon[37372]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-06T14:22:16.958 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:16 smithi142 ceph-mon[37372]: from='client.? 172.21.15.142:0/2002155790' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "d8080de9-24b0-4d10-b745-ec2a23f80e49"} : dispatch 2024-09-06T14:22:16.958 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:16 smithi142 ceph-mon[37372]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "d8080de9-24b0-4d10-b745-ec2a23f80e49"} : dispatch 2024-09-06T14:22:16.958 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:16 smithi142 ceph-mon[37372]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d8080de9-24b0-4d10-b745-ec2a23f80e49"}]': finished 2024-09-06T14:22:16.958 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:16 smithi142 ceph-mon[37372]: osdmap e22: 4 total, 3 up, 4 in 2024-09-06T14:22:16.958 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:16 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-06T14:22:16.958 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:16 smithi142 ceph-mon[29958]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-06T14:22:16.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:16 smithi142 ceph-mon[29958]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-06T14:22:16.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:16 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/2002155790' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "d8080de9-24b0-4d10-b745-ec2a23f80e49"} : dispatch 2024-09-06T14:22:16.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:16 smithi142 ceph-mon[29958]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "d8080de9-24b0-4d10-b745-ec2a23f80e49"} : dispatch 2024-09-06T14:22:16.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:16 smithi142 ceph-mon[29958]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d8080de9-24b0-4d10-b745-ec2a23f80e49"}]': finished 2024-09-06T14:22:16.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:16 smithi142 ceph-mon[29958]: osdmap e22: 4 total, 3 up, 4 in 2024-09-06T14:22:16.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:16 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-06T14:22:17.046 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:16 smithi162 ceph-mon[33787]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-06T14:22:17.046 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:16 smithi162 ceph-mon[33787]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-06T14:22:17.046 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:16 smithi162 ceph-mon[33787]: from='client.? 172.21.15.142:0/2002155790' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "d8080de9-24b0-4d10-b745-ec2a23f80e49"} : dispatch 2024-09-06T14:22:17.046 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:16 smithi162 ceph-mon[33787]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "d8080de9-24b0-4d10-b745-ec2a23f80e49"} : dispatch 2024-09-06T14:22:17.046 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:16 smithi162 ceph-mon[33787]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d8080de9-24b0-4d10-b745-ec2a23f80e49"}]': finished 2024-09-06T14:22:17.046 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:16 smithi162 ceph-mon[33787]: osdmap e22: 4 total, 3 up, 4 in 2024-09-06T14:22:17.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:16 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-06T14:22:17.957 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:17 smithi142 ceph-mon[29958]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-06T14:22:17.958 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:17 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/1189214658' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-06T14:22:17.958 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:17 smithi142 ceph-mon[37372]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-06T14:22:17.958 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:17 smithi142 ceph-mon[37372]: from='client.? 172.21.15.142:0/1189214658' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-06T14:22:18.046 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:17 smithi162 ceph-mon[33787]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-06T14:22:18.046 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:17 smithi162 ceph-mon[33787]: from='client.? 172.21.15.142:0/1189214658' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-06T14:22:20.207 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:19 smithi142 ceph-mon[29958]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-06T14:22:20.208 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:19 smithi142 ceph-mon[37372]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-06T14:22:20.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:19 smithi162 ceph-mon[33787]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-06T14:22:22.172 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:21 smithi142 ceph-mon[29958]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-06T14:22:22.172 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:21 smithi142 ceph-mon[37372]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-06T14:22:22.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:21 smithi162 ceph-mon[33787]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-06T14:22:23.958 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:23 smithi142 ceph-mon[37372]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-06T14:22:23.958 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:23 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-06T14:22:23.958 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:23 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:22:23.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:23 smithi142 ceph-mon[29958]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-06T14:22:23.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:23 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-06T14:22:23.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:23 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:22:24.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:23 smithi162 ceph-mon[33787]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-06T14:22:24.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:23 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-06T14:22:24.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:23 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:22:25.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:24 smithi142 ceph-mon[29958]: Deploying daemon osd.3 on smithi142 2024-09-06T14:22:25.208 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:24 smithi142 ceph-mon[37372]: Deploying daemon osd.3 on smithi142 2024-09-06T14:22:25.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:24 smithi162 ceph-mon[33787]: Deploying daemon osd.3 on smithi142 2024-09-06T14:22:26.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:25 smithi142 ceph-mon[29958]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-06T14:22:26.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:25 smithi142 ceph-mon[37372]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-06T14:22:26.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:25 smithi162 ceph-mon[33787]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-06T14:22:28.190 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:27 smithi142 ceph-mon[29958]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-06T14:22:28.190 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:27 smithi142 ceph-mon[37372]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-06T14:22:28.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:27 smithi162 ceph-mon[33787]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-06T14:22:30.082 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:29 smithi142 ceph-mon[29958]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-06T14:22:30.082 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:29 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-06T14:22:30.082 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:29 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:22:30.083 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:29 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:22:30.083 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:29 smithi142 ceph-mon[37372]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-06T14:22:30.083 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:29 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-06T14:22:30.083 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:29 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:22:30.083 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:29 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:22:30.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:29 smithi162 ceph-mon[33787]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-06T14:22:30.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:29 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-06T14:22:30.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:29 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:22:30.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:29 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:22:32.190 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:31 smithi142 ceph-mon[29958]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-06T14:22:32.191 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:31 smithi142 ceph-mon[37372]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-06T14:22:32.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:31 smithi162 ceph-mon[33787]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-06T14:22:32.799 INFO:teuthology.orchestra.run.smithi142.stdout:Created osd(s) 3 on host 'smithi142' 2024-09-06T14:22:32.800 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:22:32.803+0000 7f9808ff9640 1 -- 172.21.15.142:0/3059614526 <== mgr.14146 v2:172.21.15.142:6800/853729507 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f980c0632f0 con 0x7f97f40774a0 2024-09-06T14:22:32.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:22:32.805+0000 7f9811ba5640 1 -- 172.21.15.142:0/3059614526 >> v2:172.21.15.142:6800/853729507 conn(0x7f97f40774a0 msgr2=0x7f97f4079960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:22:32.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:22:32.805+0000 7f9811ba5640 1 --2- 172.21.15.142:0/3059614526 >> v2:172.21.15.142:6800/853729507 conn(0x7f97f40774a0 0x7f97f4079960 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f9800009bd0 tx=0x7f9800005f90 comp rx=0 tx=0).stop 2024-09-06T14:22:32.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:22:32.806+0000 7f9811ba5640 1 -- 172.21.15.142:0/3059614526 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f980c102760 msgr2=0x7f980c198170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:22:32.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:22:32.806+0000 7f9811ba5640 1 --2- 172.21.15.142:0/3059614526 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f980c102760 0x7f980c198170 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f97fc00c920 tx=0x7f97fc00cde0 comp rx=0 tx=0).stop 2024-09-06T14:22:32.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:22:32.806+0000 7f9811ba5640 1 -- 172.21.15.142:0/3059614526 shutdown_connections 2024-09-06T14:22:32.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:22:32.806+0000 7f9811ba5640 1 --2- 172.21.15.142:0/3059614526 >> v2:172.21.15.142:6800/853729507 conn(0x7f97f40774a0 0x7f97f4079960 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:22:32.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:22:32.806+0000 7f9811ba5640 1 --2- 172.21.15.142:0/3059614526 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f980c104280 0x7f980c199400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:22:32.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:22:32.806+0000 7f9811ba5640 1 --2- 172.21.15.142:0/3059614526 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f980c1038c0 0x7f980c1987c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:22:32.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:22:32.806+0000 7f9811ba5640 1 --2- 172.21.15.142:0/3059614526 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f980c102760 0x7f980c198170 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:22:32.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:22:32.806+0000 7f9811ba5640 1 -- 172.21.15.142:0/3059614526 >> 172.21.15.142:0/3059614526 conn(0x7f980c0fe050 msgr2=0x7f980c0ffae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:22:32.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:22:32.806+0000 7f9811ba5640 1 -- 172.21.15.142:0/3059614526 shutdown_connections 2024-09-06T14:22:32.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:22:32.806+0000 7f9811ba5640 1 -- 172.21.15.142:0/3059614526 wait complete. 2024-09-06T14:22:33.449 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:33 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:22:33.450 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:33 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:22:33.450 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:33 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:22:33.450 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:33 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:22:33.450 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:33 smithi142 ceph-mon[29958]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-06T14:22:33.450 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:33 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:22:33.450 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:33 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:22:33.450 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:33 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:22:33.450 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:33 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:22:33.451 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:33 smithi142 ceph-mon[37372]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-06T14:22:33.476 DEBUG:teuthology.orchestra.run.smithi142:osd.3> sudo journalctl -f -n 0 -u ceph-ce32462e-6c5a-11ef-bce4-c7b262605968@osd.3.service 2024-09-06T14:22:33.478 INFO:tasks.cephadm:Deploying osd.4 on smithi162 with /dev/vg_nvme/lv_4... 2024-09-06T14:22:33.479 DEBUG:teuthology.orchestra.run.smithi162:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- lvm zap /dev/vg_nvme/lv_4 2024-09-06T14:22:33.507 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:33 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:22:33.507 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:33 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:22:33.507 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:33 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:22:33.507 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:33 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:22:33.507 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:33 smithi162 ceph-mon[33787]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-06T14:22:33.733 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.b/config 2024-09-06T14:22:34.708 INFO:journalctl@ceph.osd.3.smithi142.stdout:Sep 06 14:22:34 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-3[58430]: 2024-09-06T14:22:34.367+0000 7fb0b5dfd640 -1 osd.3 0 log_to_monitors true 2024-09-06T14:22:34.987 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:34 smithi162 ceph-mon[33787]: from='osd.3 v2:172.21.15.142:6813/2917907718' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-06T14:22:34.988 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:34 smithi162 ceph-mon[33787]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-06T14:22:35.056 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:34 smithi142 ceph-mon[29958]: from='osd.3 v2:172.21.15.142:6813/2917907718' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-06T14:22:35.057 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:34 smithi142 ceph-mon[29958]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-06T14:22:35.057 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:34 smithi142 ceph-mon[37372]: from='osd.3 v2:172.21.15.142:6813/2917907718' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-06T14:22:35.057 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:34 smithi142 ceph-mon[37372]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-06T14:22:36.062 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-06T14:22:36.073 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:35 smithi142 ceph-mon[29958]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-06T14:22:36.073 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:35 smithi142 ceph-mon[29958]: osdmap e23: 4 total, 3 up, 4 in 2024-09-06T14:22:36.073 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:35 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-06T14:22:36.073 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:35 smithi142 ceph-mon[29958]: from='osd.3 v2:172.21.15.142:6813/2917907718' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi142", "root=default"]} : dispatch 2024-09-06T14:22:36.074 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:35 smithi142 ceph-mon[29958]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi142", "root=default"]} : dispatch 2024-09-06T14:22:36.074 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:35 smithi142 ceph-mon[29958]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-06T14:22:36.074 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:35 smithi142 ceph-mon[37372]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-06T14:22:36.074 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:35 smithi142 ceph-mon[37372]: osdmap e23: 4 total, 3 up, 4 in 2024-09-06T14:22:36.074 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:35 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-06T14:22:36.074 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:35 smithi142 ceph-mon[37372]: from='osd.3 v2:172.21.15.142:6813/2917907718' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi142", "root=default"]} : dispatch 2024-09-06T14:22:36.074 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:35 smithi142 ceph-mon[37372]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi142", "root=default"]} : dispatch 2024-09-06T14:22:36.074 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:35 smithi142 ceph-mon[37372]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-06T14:22:36.086 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:35 smithi162 ceph-mon[33787]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-06T14:22:36.086 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:35 smithi162 ceph-mon[33787]: osdmap e23: 4 total, 3 up, 4 in 2024-09-06T14:22:36.086 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:35 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-06T14:22:36.086 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:35 smithi162 ceph-mon[33787]: from='osd.3 v2:172.21.15.142:6813/2917907718' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi142", "root=default"]} : dispatch 2024-09-06T14:22:36.087 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:35 smithi162 ceph-mon[33787]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi142", "root=default"]} : dispatch 2024-09-06T14:22:36.087 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:35 smithi162 ceph-mon[33787]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-06T14:22:36.088 DEBUG:teuthology.orchestra.run.smithi162:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph orch daemon add osd smithi162:vg_nvme/lv_4 2024-09-06T14:22:36.379 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.b/config 2024-09-06T14:22:36.405 INFO:journalctl@ceph.osd.3.smithi142.stdout:Sep 06 14:22:36 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-3[58430]: 2024-09-06T14:22:36.244+0000 7fb0b1d80640 -1 osd.3 0 waiting for initial osdmap 2024-09-06T14:22:36.405 INFO:journalctl@ceph.osd.3.smithi142.stdout:Sep 06 14:22:36 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-3[58430]: 2024-09-06T14:22:36.250+0000 7fb0a9b9f640 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-06T14:22:36.779 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:22:36.778+0000 7f1e026ab640 1 -- 172.21.15.162:0/3071622014 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1dfc103190 msgr2=0x7f1dfc109b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:22:36.779 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:22:36.778+0000 7f1e026ab640 1 --2- 172.21.15.162:0/3071622014 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1dfc103190 0x7f1dfc109b10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1df4009cc0 tx=0x7f1df402a8a0 comp rx=0 tx=0).stop 2024-09-06T14:22:36.779 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:22:36.778+0000 7f1e026ab640 1 -- 172.21.15.162:0/3071622014 shutdown_connections 2024-09-06T14:22:36.779 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:22:36.778+0000 7f1e026ab640 1 --2- 172.21.15.162:0/3071622014 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1dfc103190 0x7f1dfc109b10 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:22:36.779 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:22:36.778+0000 7f1e026ab640 1 --2- 172.21.15.162:0/3071622014 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1dfc1027d0 0x7f1dfc102c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:22:36.779 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:22:36.778+0000 7f1e026ab640 1 --2- 172.21.15.162:0/3071622014 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1dfc101670 0x7f1dfc101a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:22:36.779 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:22:36.778+0000 7f1e026ab640 1 -- 172.21.15.162:0/3071622014 >> 172.21.15.162:0/3071622014 conn(0x7f1dfc0fcf60 msgr2=0x7f1dfc0ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:22:36.779 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:22:36.778+0000 7f1e026ab640 1 -- 172.21.15.162:0/3071622014 shutdown_connections 2024-09-06T14:22:36.779 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:22:36.778+0000 7f1e026ab640 1 -- 172.21.15.162:0/3071622014 wait complete. 2024-09-06T14:22:36.779 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:22:36.779+0000 7f1e026ab640 1 Processor -- start 2024-09-06T14:22:36.780 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:22:36.779+0000 7f1e026ab640 1 -- start start 2024-09-06T14:22:36.780 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:22:36.779+0000 7f1e026ab640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1dfc101670 0x7f1dfc105430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:22:36.780 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:22:36.779+0000 7f1e026ab640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1dfc1027d0 0x7f1dfc103ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:22:36.780 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:22:36.779+0000 7f1e026ab640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1dfc103190 0x7f1dfc1041e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:22:36.780 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:22:36.779+0000 7f1e026ab640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f1dfc10c960 con 0x7f1dfc1027d0 2024-09-06T14:22:36.780 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:22:36.779+0000 7f1e026ab640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1dfc10c7e0 con 0x7f1dfc103190 2024-09-06T14:22:36.780 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:22:36.779+0000 7f1e026ab640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7f1dfc10cae0 con 0x7f1dfc101670 2024-09-06T14:22:36.780 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:22:36.779+0000 7f1e00c48640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1dfc103190 0x7f1dfc1041e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:22:36.780 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:22:36.779+0000 7f1e00c48640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1dfc103190 0x7f1dfc1041e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:44478/0 (socket says 172.21.15.162:44478) 2024-09-06T14:22:36.780 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:22:36.779+0000 7f1e00c48640 1 -- 172.21.15.162:0/4281596295 learned_addr learned my addr 172.21.15.162:0/4281596295 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-09-06T14:22:36.780 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:22:36.779+0000 7f1dfbfff640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1dfc101670 0x7f1dfc105430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:22:36.780 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:22:36.779+0000 7f1df3fff640 1 --2- 172.21.15.162:0/4281596295 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1dfc1027d0 0x7f1dfc103ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:22:36.780 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:22:36.779+0000 7f1e00c48640 1 -- 172.21.15.162:0/4281596295 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1dfc101670 msgr2=0x7f1dfc105430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:22:36.781 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:22:36.779+0000 7f1e00c48640 1 --2- 172.21.15.162:0/4281596295 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1dfc101670 0x7f1dfc105430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:22:36.781 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:22:36.779+0000 7f1e00c48640 1 -- 172.21.15.162:0/4281596295 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1dfc1027d0 msgr2=0x7f1dfc103ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:22:36.781 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:22:36.779+0000 7f1e00c48640 1 --2- 172.21.15.162:0/4281596295 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1dfc1027d0 0x7f1dfc103ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:22:36.781 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:22:36.779+0000 7f1e00c48640 1 -- 172.21.15.162:0/4281596295 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1dfc104aa0 con 0x7f1dfc103190 2024-09-06T14:22:36.781 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:22:36.780+0000 7f1e00c48640 1 --2- 172.21.15.162:0/4281596295 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1dfc103190 0x7f1dfc1041e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f1df4005b90 tx=0x7f1df4009310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:22:36.781 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:22:36.780+0000 7f1df9ffb640 1 -- 172.21.15.162:0/4281596295 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1df4004c90 con 0x7f1dfc103190 2024-09-06T14:22:36.781 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:22:36.780+0000 7f1e026ab640 1 -- 172.21.15.162:0/4281596295 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1dfc1a2360 con 0x7f1dfc103190 2024-09-06T14:22:36.781 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:22:36.780+0000 7f1df9ffb640 1 -- 172.21.15.162:0/4281596295 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7f1df4004e30 con 0x7f1dfc103190 2024-09-06T14:22:36.781 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:22:36.780+0000 7f1df9ffb640 1 -- 172.21.15.162:0/4281596295 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1df4005720 con 0x7f1dfc103190 2024-09-06T14:22:36.781 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:22:36.780+0000 7f1e026ab640 1 -- 172.21.15.162:0/4281596295 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1dfc1a2870 con 0x7f1dfc103190 2024-09-06T14:22:36.781 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:22:36.781+0000 7f1e026ab640 1 -- 172.21.15.162:0/4281596295 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1dfc101a70 con 0x7f1dfc103190 2024-09-06T14:22:36.781 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:22:36.782+0000 7f1df9ffb640 1 -- 172.21.15.162:0/4281596295 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 15) ==== 99786+0+0 (secure 0 0 0) 0x7f1df4030870 con 0x7f1dfc103190 2024-09-06T14:22:36.781 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:22:36.782+0000 7f1df9ffb640 1 --2- 172.21.15.162:0/4281596295 >> v2:172.21.15.142:6800/853729507 conn(0x7f1de8077530 0x7f1de80799f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:22:36.782 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:22:36.782+0000 7f1df9ffb640 1 -- 172.21.15.162:0/4281596295 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 2577+0+0 (secure 0 0 0) 0x7f1df40b94c0 con 0x7f1dfc103190 2024-09-06T14:22:36.782 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:22:36.782+0000 7f1dfbfff640 1 --2- 172.21.15.162:0/4281596295 >> v2:172.21.15.142:6800/853729507 conn(0x7f1de8077530 0x7f1de80799f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:22:36.782 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:22:36.783+0000 7f1dfbfff640 1 --2- 172.21.15.162:0/4281596295 >> v2:172.21.15.142:6800/853729507 conn(0x7f1de8077530 0x7f1de80799f0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f1dec009bd0 tx=0x7f1dec005f90 comp rx=0 tx=0).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:22:36.782 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:22:36.786+0000 7f1df9ffb640 1 -- 172.21.15.162:0/4281596295 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7f1df4082e30 con 0x7f1dfc103190 2024-09-06T14:22:36.952 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:22:36.956+0000 7f1e026ab640 1 -- 172.21.15.162:0/4281596295 --> v2:172.21.15.142:6800/853729507 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f1dfc0632f0 con 0x7f1de8077530 2024-09-06T14:22:36.952 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:36 smithi162 ceph-mon[33787]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]': finished 2024-09-06T14:22:36.952 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:36 smithi162 ceph-mon[33787]: osdmap e24: 4 total, 3 up, 4 in 2024-09-06T14:22:36.952 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:36 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-06T14:22:36.952 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:36 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-06T14:22:36.952 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:36 smithi162 ceph-mon[33787]: from='osd.3 ' entity='osd.3' 2024-09-06T14:22:36.953 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:36 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:22:36.953 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:36 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:22:36.953 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:36 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-06T14:22:36.953 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:36 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-06T14:22:36.953 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:36 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-06T14:22:36.953 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:36 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-06T14:22:36.953 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:36 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:22:36.953 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:36 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:22:36.953 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:36 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-06T14:22:36.953 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:36 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:22:36.953 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:36 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-06T14:22:37.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:36 smithi142 ceph-mon[29958]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]': finished 2024-09-06T14:22:37.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:36 smithi142 ceph-mon[29958]: osdmap e24: 4 total, 3 up, 4 in 2024-09-06T14:22:37.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:36 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-06T14:22:37.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:36 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-06T14:22:37.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:36 smithi142 ceph-mon[29958]: from='osd.3 ' entity='osd.3' 2024-09-06T14:22:37.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:36 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:22:37.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:36 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:22:37.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:36 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-06T14:22:37.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:36 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-06T14:22:37.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:36 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-06T14:22:37.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:36 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-06T14:22:37.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:36 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:22:37.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:36 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:22:37.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:36 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-06T14:22:37.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:36 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:22:37.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:36 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-06T14:22:37.210 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:36 smithi142 ceph-mon[37372]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]': finished 2024-09-06T14:22:37.210 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:36 smithi142 ceph-mon[37372]: osdmap e24: 4 total, 3 up, 4 in 2024-09-06T14:22:37.210 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:36 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-06T14:22:37.210 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:36 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-06T14:22:37.210 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:36 smithi142 ceph-mon[37372]: from='osd.3 ' entity='osd.3' 2024-09-06T14:22:37.210 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:36 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:22:37.210 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:36 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:22:37.210 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:36 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-06T14:22:37.210 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:36 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-06T14:22:37.210 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:36 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-06T14:22:37.210 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:36 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-06T14:22:37.210 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:36 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:22:37.211 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:36 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:22:37.211 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:36 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-06T14:22:37.211 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:36 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:22:37.211 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:36 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-06T14:22:38.046 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:37 smithi162 ceph-mon[33787]: purged_snaps scrub starts 2024-09-06T14:22:38.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:37 smithi162 ceph-mon[33787]: purged_snaps scrub ok 2024-09-06T14:22:38.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:37 smithi162 ceph-mon[33787]: Detected new or changed devices on smithi142 2024-09-06T14:22:38.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:37 smithi162 ceph-mon[33787]: Adjusting osd_memory_target on smithi142 to 4016M 2024-09-06T14:22:38.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:37 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-06T14:22:38.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:37 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:22:38.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:37 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-06T14:22:38.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:37 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:22:38.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:37 smithi162 ceph-mon[33787]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-06T14:22:38.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:37 smithi162 ceph-mon[33787]: from='client.24187 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-06T14:22:38.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:37 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-06T14:22:38.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:37 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-06T14:22:38.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:37 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:22:38.048 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:37 smithi162 ceph-mon[33787]: osd.3 v2:172.21.15.142:6813/2917907718 boot 2024-09-06T14:22:38.048 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:37 smithi162 ceph-mon[33787]: osdmap e25: 4 total, 4 up, 4 in 2024-09-06T14:22:38.048 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:37 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-06T14:22:38.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:37 smithi142 ceph-mon[29958]: purged_snaps scrub starts 2024-09-06T14:22:38.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:37 smithi142 ceph-mon[29958]: purged_snaps scrub ok 2024-09-06T14:22:38.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:37 smithi142 ceph-mon[29958]: Detected new or changed devices on smithi142 2024-09-06T14:22:38.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:37 smithi142 ceph-mon[29958]: Adjusting osd_memory_target on smithi142 to 4016M 2024-09-06T14:22:38.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:37 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-06T14:22:38.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:37 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:22:38.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:37 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-06T14:22:38.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:37 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:22:38.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:37 smithi142 ceph-mon[29958]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-06T14:22:38.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:37 smithi142 ceph-mon[29958]: from='client.24187 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-06T14:22:38.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:37 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-06T14:22:38.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:37 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-06T14:22:38.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:37 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:22:38.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:37 smithi142 ceph-mon[29958]: osd.3 v2:172.21.15.142:6813/2917907718 boot 2024-09-06T14:22:38.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:37 smithi142 ceph-mon[29958]: osdmap e25: 4 total, 4 up, 4 in 2024-09-06T14:22:38.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:37 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-06T14:22:38.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:37 smithi142 ceph-mon[37372]: purged_snaps scrub starts 2024-09-06T14:22:38.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:37 smithi142 ceph-mon[37372]: purged_snaps scrub ok 2024-09-06T14:22:38.210 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:37 smithi142 ceph-mon[37372]: Detected new or changed devices on smithi142 2024-09-06T14:22:38.210 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:37 smithi142 ceph-mon[37372]: Adjusting osd_memory_target on smithi142 to 4016M 2024-09-06T14:22:38.210 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:37 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-06T14:22:38.210 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:37 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:22:38.210 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:37 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-06T14:22:38.210 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:37 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:22:38.210 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:37 smithi142 ceph-mon[37372]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-06T14:22:38.210 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:37 smithi142 ceph-mon[37372]: from='client.24187 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-06T14:22:38.210 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:37 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-06T14:22:38.210 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:37 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-06T14:22:38.210 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:37 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:22:38.210 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:37 smithi142 ceph-mon[37372]: osd.3 v2:172.21.15.142:6813/2917907718 boot 2024-09-06T14:22:38.210 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:37 smithi142 ceph-mon[37372]: osdmap e25: 4 total, 4 up, 4 in 2024-09-06T14:22:38.211 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:37 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-06T14:22:39.539 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:39 smithi162 ceph-mon[33787]: osdmap e26: 4 total, 4 up, 4 in 2024-09-06T14:22:39.539 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:39 smithi162 ceph-mon[33787]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-06T14:22:39.540 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:39 smithi162 ceph-mon[33787]: from='client.? 172.21.15.162:0/1310495175' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "d8c6b71c-01ef-44a5-b78b-49ecce11f357"} : dispatch 2024-09-06T14:22:39.540 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:39 smithi162 ceph-mon[33787]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "d8c6b71c-01ef-44a5-b78b-49ecce11f357"} : dispatch 2024-09-06T14:22:39.540 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:39 smithi162 ceph-mon[33787]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d8c6b71c-01ef-44a5-b78b-49ecce11f357"}]': finished 2024-09-06T14:22:39.540 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:39 smithi162 ceph-mon[33787]: osdmap e27: 5 total, 4 up, 5 in 2024-09-06T14:22:39.540 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:39 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-06T14:22:39.708 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:39 smithi142 ceph-mon[29958]: osdmap e26: 4 total, 4 up, 4 in 2024-09-06T14:22:39.708 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:39 smithi142 ceph-mon[29958]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-06T14:22:39.708 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:39 smithi142 ceph-mon[29958]: from='client.? 172.21.15.162:0/1310495175' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "d8c6b71c-01ef-44a5-b78b-49ecce11f357"} : dispatch 2024-09-06T14:22:39.708 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:39 smithi142 ceph-mon[29958]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "d8c6b71c-01ef-44a5-b78b-49ecce11f357"} : dispatch 2024-09-06T14:22:39.708 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:39 smithi142 ceph-mon[29958]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d8c6b71c-01ef-44a5-b78b-49ecce11f357"}]': finished 2024-09-06T14:22:39.708 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:39 smithi142 ceph-mon[29958]: osdmap e27: 5 total, 4 up, 5 in 2024-09-06T14:22:39.708 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:39 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-06T14:22:39.709 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:39 smithi142 ceph-mon[37372]: osdmap e26: 4 total, 4 up, 4 in 2024-09-06T14:22:39.709 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:39 smithi142 ceph-mon[37372]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-06T14:22:39.709 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:39 smithi142 ceph-mon[37372]: from='client.? 172.21.15.162:0/1310495175' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "d8c6b71c-01ef-44a5-b78b-49ecce11f357"} : dispatch 2024-09-06T14:22:39.709 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:39 smithi142 ceph-mon[37372]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "d8c6b71c-01ef-44a5-b78b-49ecce11f357"} : dispatch 2024-09-06T14:22:39.709 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:39 smithi142 ceph-mon[37372]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d8c6b71c-01ef-44a5-b78b-49ecce11f357"}]': finished 2024-09-06T14:22:39.709 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:39 smithi142 ceph-mon[37372]: osdmap e27: 5 total, 4 up, 5 in 2024-09-06T14:22:39.709 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:39 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-06T14:22:40.546 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:40 smithi162 ceph-mon[33787]: from='client.? 172.21.15.162:0/3239255838' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-06T14:22:40.708 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:40 smithi142 ceph-mon[29958]: from='client.? 172.21.15.162:0/3239255838' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-06T14:22:40.708 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:40 smithi142 ceph-mon[37372]: from='client.? 172.21.15.162:0/3239255838' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-06T14:22:41.546 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:41 smithi162 ceph-mon[33787]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-06T14:22:41.708 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:41 smithi142 ceph-mon[29958]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-06T14:22:41.708 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:41 smithi142 ceph-mon[37372]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-06T14:22:44.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:43 smithi142 ceph-mon[29958]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-06T14:22:44.208 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:43 smithi142 ceph-mon[37372]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-06T14:22:44.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:43 smithi162 ceph-mon[33787]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-06T14:22:46.046 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:45 smithi162 ceph-mon[33787]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-06T14:22:46.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:45 smithi142 ceph-mon[29958]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-06T14:22:46.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:45 smithi142 ceph-mon[37372]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-06T14:22:47.046 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:46 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-06T14:22:47.046 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:46 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:22:47.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:46 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-06T14:22:47.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:46 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:22:47.208 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:46 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-06T14:22:47.208 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:46 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:22:48.186 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:47 smithi162 ceph-mon[33787]: Deploying daemon osd.4 on smithi162 2024-09-06T14:22:48.186 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:47 smithi162 ceph-mon[33787]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-06T14:22:48.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:47 smithi142 ceph-mon[29958]: Deploying daemon osd.4 on smithi162 2024-09-06T14:22:48.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:47 smithi142 ceph-mon[29958]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-06T14:22:48.208 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:47 smithi142 ceph-mon[37372]: Deploying daemon osd.4 on smithi162 2024-09-06T14:22:48.208 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:47 smithi142 ceph-mon[37372]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-06T14:22:50.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:49 smithi142 ceph-mon[29958]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-06T14:22:50.208 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:49 smithi142 ceph-mon[37372]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-06T14:22:50.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:49 smithi162 ceph-mon[33787]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-06T14:22:52.132 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:51 smithi162 ceph-mon[33787]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-06T14:22:52.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:51 smithi142 ceph-mon[29958]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-06T14:22:52.208 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:51 smithi142 ceph-mon[37372]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-06T14:22:53.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:52 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-06T14:22:53.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:52 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:22:53.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:52 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:22:53.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:52 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-06T14:22:53.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:52 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:22:53.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:52 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:22:53.270 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:52 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-06T14:22:53.270 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:52 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:22:53.270 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:52 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:22:54.115 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:53 smithi162 ceph-mon[33787]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-06T14:22:54.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:53 smithi142 ceph-mon[29958]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-06T14:22:54.208 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:53 smithi142 ceph-mon[37372]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-06T14:22:56.046 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:55 smithi162 ceph-mon[33787]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-06T14:22:56.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:55 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:22:56.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:55 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:22:56.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:55 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:22:56.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:55 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-06T14:22:56.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:55 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:22:56.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:55 smithi142 ceph-mon[29958]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-06T14:22:56.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:55 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:22:56.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:55 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:22:56.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:55 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:22:56.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:55 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-06T14:22:56.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:55 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:22:56.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:55 smithi142 ceph-mon[37372]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-06T14:22:56.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:55 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:22:56.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:55 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:22:56.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:55 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:22:56.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:55 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-06T14:22:56.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:55 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:22:56.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:22:56.460+0000 7f1df9ffb640 1 -- 172.21.15.162:0/4281596295 <== mgr.14146 v2:172.21.15.142:6800/853729507 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f1dfc0632f0 con 0x7f1de8077530 2024-09-06T14:22:56.458 INFO:teuthology.orchestra.run.smithi162.stdout:Created osd(s) 4 on host 'smithi162' 2024-09-06T14:22:56.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:22:56.462+0000 7f1e026ab640 1 -- 172.21.15.162:0/4281596295 >> v2:172.21.15.142:6800/853729507 conn(0x7f1de8077530 msgr2=0x7f1de80799f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:22:56.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:22:56.462+0000 7f1e026ab640 1 --2- 172.21.15.162:0/4281596295 >> v2:172.21.15.142:6800/853729507 conn(0x7f1de8077530 0x7f1de80799f0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f1dec009bd0 tx=0x7f1dec005f90 comp rx=0 tx=0).stop 2024-09-06T14:22:56.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:22:56.462+0000 7f1e026ab640 1 -- 172.21.15.162:0/4281596295 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1dfc103190 msgr2=0x7f1dfc1041e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:22:56.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:22:56.462+0000 7f1e026ab640 1 --2- 172.21.15.162:0/4281596295 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1dfc103190 0x7f1dfc1041e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f1df4005b90 tx=0x7f1df4009310 comp rx=0 tx=0).stop 2024-09-06T14:22:56.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:22:56.462+0000 7f1e026ab640 1 -- 172.21.15.162:0/4281596295 shutdown_connections 2024-09-06T14:22:56.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:22:56.462+0000 7f1e026ab640 1 --2- 172.21.15.162:0/4281596295 >> v2:172.21.15.142:6800/853729507 conn(0x7f1de8077530 0x7f1de80799f0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:22:56.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:22:56.463+0000 7f1e026ab640 1 --2- 172.21.15.162:0/4281596295 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1dfc103190 0x7f1dfc1041e0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:22:56.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:22:56.463+0000 7f1e026ab640 1 --2- 172.21.15.162:0/4281596295 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1dfc1027d0 0x7f1dfc103ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:22:56.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:22:56.463+0000 7f1e026ab640 1 --2- 172.21.15.162:0/4281596295 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1dfc101670 0x7f1dfc105430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:22:56.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:22:56.463+0000 7f1e026ab640 1 -- 172.21.15.162:0/4281596295 >> 172.21.15.162:0/4281596295 conn(0x7f1dfc0fcf60 msgr2=0x7f1dfc06ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:22:56.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:22:56.463+0000 7f1e026ab640 1 -- 172.21.15.162:0/4281596295 shutdown_connections 2024-09-06T14:22:56.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:22:56.463+0000 7f1e026ab640 1 -- 172.21.15.162:0/4281596295 wait complete. 2024-09-06T14:22:57.104 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:56 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-06T14:22:57.104 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:56 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:22:57.104 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:56 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:22:57.131 DEBUG:teuthology.orchestra.run.smithi162:osd.4> sudo journalctl -f -n 0 -u ceph-ce32462e-6c5a-11ef-bce4-c7b262605968@osd.4.service 2024-09-06T14:22:57.133 INFO:tasks.cephadm:Deploying osd.5 on smithi162 with /dev/vg_nvme/lv_3... 2024-09-06T14:22:57.133 DEBUG:teuthology.orchestra.run.smithi162:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- lvm zap /dev/vg_nvme/lv_3 2024-09-06T14:22:57.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:56 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-06T14:22:57.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:56 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:22:57.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:56 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:22:57.208 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:56 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-06T14:22:57.208 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:56 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:22:57.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:56 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:22:58.009 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:57 smithi162 ceph-mon[33787]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-06T14:22:58.009 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:57 smithi162 ceph-mon[33787]: from='osd.4 v2:172.21.15.162:6800/2665827777' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-06T14:22:58.010 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:57 smithi162 ceph-mon[33787]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-06T14:22:58.010 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 06 14:22:57 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-4[38539]: 2024-09-06T14:22:57.746+0000 7f36afb21640 -1 osd.4 0 log_to_monitors true 2024-09-06T14:22:58.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:57 smithi142 ceph-mon[29958]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-06T14:22:58.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:57 smithi142 ceph-mon[29958]: from='osd.4 v2:172.21.15.162:6800/2665827777' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-06T14:22:58.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:57 smithi142 ceph-mon[29958]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-06T14:22:58.208 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:57 smithi142 ceph-mon[37372]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-06T14:22:58.208 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:57 smithi142 ceph-mon[37372]: from='osd.4 v2:172.21.15.162:6800/2665827777' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-06T14:22:58.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:57 smithi142 ceph-mon[37372]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-06T14:22:58.734 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.b/config 2024-09-06T14:22:59.737 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:59 smithi162 ceph-mon[33787]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-06T14:22:59.737 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:59 smithi162 ceph-mon[33787]: from='osd.4 v2:172.21.15.162:6800/2665827777' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-06T14:22:59.737 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:59 smithi162 ceph-mon[33787]: osdmap e28: 5 total, 4 up, 5 in 2024-09-06T14:22:59.737 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:59 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-06T14:22:59.737 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:59 smithi162 ceph-mon[33787]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-06T14:22:59.737 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:22:59 smithi162 ceph-mon[33787]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-06T14:22:59.738 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 06 14:22:59 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-4[38539]: 2024-09-06T14:22:59.467+0000 7f36abaa4640 -1 osd.4 0 waiting for initial osdmap 2024-09-06T14:22:59.738 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 06 14:22:59 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-4[38539]: 2024-09-06T14:22:59.471+0000 7f36a78cb640 -1 osd.4 29 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-06T14:22:59.958 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:59 smithi142 ceph-mon[29958]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-06T14:22:59.958 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:59 smithi142 ceph-mon[29958]: from='osd.4 v2:172.21.15.162:6800/2665827777' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-06T14:22:59.958 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:59 smithi142 ceph-mon[29958]: osdmap e28: 5 total, 4 up, 5 in 2024-09-06T14:22:59.958 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:59 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-06T14:22:59.958 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:59 smithi142 ceph-mon[29958]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-06T14:22:59.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:22:59 smithi142 ceph-mon[29958]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-06T14:22:59.959 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:59 smithi142 ceph-mon[37372]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-06T14:22:59.959 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:59 smithi142 ceph-mon[37372]: from='osd.4 v2:172.21.15.162:6800/2665827777' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-06T14:22:59.959 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:59 smithi142 ceph-mon[37372]: osdmap e28: 5 total, 4 up, 5 in 2024-09-06T14:22:59.959 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:59 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-06T14:22:59.959 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:59 smithi142 ceph-mon[37372]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-06T14:22:59.959 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:22:59 smithi142 ceph-mon[37372]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-06T14:23:00.682 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:00 smithi162 ceph-mon[33787]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-09-06T14:23:00.682 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:00 smithi162 ceph-mon[33787]: osdmap e29: 5 total, 4 up, 5 in 2024-09-06T14:23:00.682 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:00 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-06T14:23:00.682 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:00 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-06T14:23:00.682 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:00 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:23:00.682 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:00 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:23:00.682 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:00 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-06T14:23:00.682 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:00 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:23:00.682 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:00 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:23:00.682 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:00 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-06T14:23:00.683 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:00 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:23:00.961 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:00 smithi142 ceph-mon[29958]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-09-06T14:23:00.961 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:00 smithi142 ceph-mon[29958]: osdmap e29: 5 total, 4 up, 5 in 2024-09-06T14:23:00.962 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:00 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-06T14:23:00.962 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:00 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-06T14:23:00.962 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:00 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:23:00.962 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:00 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:23:00.962 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:00 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-06T14:23:00.962 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:00 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:23:00.962 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:00 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:23:00.962 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:00 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-06T14:23:00.962 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:00 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:23:00.962 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:00 smithi142 ceph-mon[37372]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-09-06T14:23:00.962 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:00 smithi142 ceph-mon[37372]: osdmap e29: 5 total, 4 up, 5 in 2024-09-06T14:23:00.963 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:00 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-06T14:23:00.963 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:00 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-06T14:23:00.963 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:00 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:23:00.963 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:00 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:23:00.963 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:00 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-06T14:23:00.963 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:00 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:23:00.963 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:00 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:23:00.963 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:00 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-06T14:23:00.963 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:00 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:23:01.797 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:01 smithi162 ceph-mon[33787]: purged_snaps scrub starts 2024-09-06T14:23:01.797 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:01 smithi162 ceph-mon[33787]: purged_snaps scrub ok 2024-09-06T14:23:01.797 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:01 smithi162 ceph-mon[33787]: Detected new or changed devices on smithi162 2024-09-06T14:23:01.797 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:01 smithi162 ceph-mon[33787]: Adjusting osd_memory_target on smithi162 to 17088M 2024-09-06T14:23:01.797 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:01 smithi162 ceph-mon[33787]: osd.4 v2:172.21.15.162:6800/2665827777 boot 2024-09-06T14:23:01.797 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:01 smithi162 ceph-mon[33787]: osdmap e30: 5 total, 5 up, 5 in 2024-09-06T14:23:01.797 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:01 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-06T14:23:01.797 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:01 smithi162 ceph-mon[33787]: pgmap v111: 1 pgs: 1 remapped+peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-06T14:23:01.797 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:01 smithi162 ceph-mon[33787]: osdmap e31: 5 total, 5 up, 5 in 2024-09-06T14:23:01.841 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-06T14:23:01.866 DEBUG:teuthology.orchestra.run.smithi162:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph orch daemon add osd smithi162:vg_nvme/lv_3 2024-09-06T14:23:01.958 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:01 smithi142 ceph-mon[29958]: purged_snaps scrub starts 2024-09-06T14:23:01.958 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:01 smithi142 ceph-mon[29958]: purged_snaps scrub ok 2024-09-06T14:23:01.958 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:01 smithi142 ceph-mon[29958]: Detected new or changed devices on smithi162 2024-09-06T14:23:01.958 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:01 smithi142 ceph-mon[29958]: Adjusting osd_memory_target on smithi162 to 17088M 2024-09-06T14:23:01.958 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:01 smithi142 ceph-mon[29958]: osd.4 v2:172.21.15.162:6800/2665827777 boot 2024-09-06T14:23:01.958 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:01 smithi142 ceph-mon[29958]: osdmap e30: 5 total, 5 up, 5 in 2024-09-06T14:23:01.958 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:01 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-06T14:23:01.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:01 smithi142 ceph-mon[29958]: pgmap v111: 1 pgs: 1 remapped+peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-06T14:23:01.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:01 smithi142 ceph-mon[29958]: osdmap e31: 5 total, 5 up, 5 in 2024-09-06T14:23:01.959 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:01 smithi142 ceph-mon[37372]: purged_snaps scrub starts 2024-09-06T14:23:01.959 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:01 smithi142 ceph-mon[37372]: purged_snaps scrub ok 2024-09-06T14:23:01.959 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:01 smithi142 ceph-mon[37372]: Detected new or changed devices on smithi162 2024-09-06T14:23:01.959 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:01 smithi142 ceph-mon[37372]: Adjusting osd_memory_target on smithi162 to 17088M 2024-09-06T14:23:01.959 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:01 smithi142 ceph-mon[37372]: osd.4 v2:172.21.15.162:6800/2665827777 boot 2024-09-06T14:23:01.959 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:01 smithi142 ceph-mon[37372]: osdmap e30: 5 total, 5 up, 5 in 2024-09-06T14:23:01.959 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:01 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-06T14:23:01.959 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:01 smithi142 ceph-mon[37372]: pgmap v111: 1 pgs: 1 remapped+peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-06T14:23:01.959 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:01 smithi142 ceph-mon[37372]: osdmap e31: 5 total, 5 up, 5 in 2024-09-06T14:23:02.111 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.b/config 2024-09-06T14:23:02.573 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:02.561+0000 7fd7eefa7640 1 -- 172.21.15.162:0/1470282371 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd7e806bdf0 msgr2=0x7fd7e8105360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:23:02.573 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:02.561+0000 7fd7eefa7640 1 --2- 172.21.15.162:0/1470282371 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd7e806bdf0 0x7fd7e8105360 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fd7dc009900 tx=0x7fd7dc02a410 comp rx=0 tx=0).stop 2024-09-06T14:23:02.573 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:02.561+0000 7fd7eefa7640 1 -- 172.21.15.162:0/1470282371 shutdown_connections 2024-09-06T14:23:02.574 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:02.561+0000 7fd7eefa7640 1 --2- 172.21.15.162:0/1470282371 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd7e81084a0 0x7fd7e810a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:23:02.574 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:02.561+0000 7fd7eefa7640 1 --2- 172.21.15.162:0/1470282371 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd7e81058f0 0x7fd7e8107d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:23:02.574 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:02.561+0000 7fd7eefa7640 1 --2- 172.21.15.162:0/1470282371 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd7e806bdf0 0x7fd7e8105360 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:23:02.574 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:02.561+0000 7fd7eefa7640 1 -- 172.21.15.162:0/1470282371 >> 172.21.15.162:0/1470282371 conn(0x7fd7e80fcf30 msgr2=0x7fd7e80ff390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:23:02.574 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:02.561+0000 7fd7eefa7640 1 -- 172.21.15.162:0/1470282371 shutdown_connections 2024-09-06T14:23:02.574 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:02.561+0000 7fd7eefa7640 1 -- 172.21.15.162:0/1470282371 wait complete. 2024-09-06T14:23:02.574 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:02.562+0000 7fd7eefa7640 1 Processor -- start 2024-09-06T14:23:02.574 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:02.562+0000 7fd7eefa7640 1 -- start start 2024-09-06T14:23:02.574 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:02.562+0000 7fd7eefa7640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd7e806bdf0 0x7fd7e8102a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:23:02.574 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:02.562+0000 7fd7eefa7640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd7e81058f0 0x7fd7e8101300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:23:02.574 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:02.562+0000 7fd7eefa7640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd7e81084a0 0x7fd7e8101840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:23:02.574 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:02.562+0000 7fd7eefa7640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fd7e810dab0 con 0x7fd7e81084a0 2024-09-06T14:23:02.574 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:02.562+0000 7fd7eefa7640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd7e810d930 con 0x7fd7e806bdf0 2024-09-06T14:23:02.574 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:02.562+0000 7fd7eefa7640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7fd7e810dc30 con 0x7fd7e81058f0 2024-09-06T14:23:02.575 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:02.562+0000 7fd7ecd43640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd7e806bdf0 0x7fd7e8102a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:23:02.575 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:02.562+0000 7fd7ecd43640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd7e806bdf0 0x7fd7e8102a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:43748/0 (socket says 172.21.15.162:43748) 2024-09-06T14:23:02.575 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:02.562+0000 7fd7ecd43640 1 -- 172.21.15.162:0/2364549495 learned_addr learned my addr 172.21.15.162:0/2364549495 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-09-06T14:23:02.575 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:02.562+0000 7fd7e7fff640 1 --2- 172.21.15.162:0/2364549495 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd7e81058f0 0x7fd7e8101300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:23:02.575 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:02.562+0000 7fd7ed544640 1 --2- 172.21.15.162:0/2364549495 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd7e81084a0 0x7fd7e8101840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:23:02.575 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:02.562+0000 7fd7ecd43640 1 -- 172.21.15.162:0/2364549495 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd7e81058f0 msgr2=0x7fd7e8101300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:23:02.575 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:02.562+0000 7fd7ecd43640 1 --2- 172.21.15.162:0/2364549495 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd7e81058f0 0x7fd7e8101300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:23:02.575 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:02.562+0000 7fd7ecd43640 1 -- 172.21.15.162:0/2364549495 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd7e81084a0 msgr2=0x7fd7e8101840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:23:02.575 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:02.562+0000 7fd7ecd43640 1 --2- 172.21.15.162:0/2364549495 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd7e81084a0 0x7fd7e8101840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:23:02.575 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:02.562+0000 7fd7ecd43640 1 -- 172.21.15.162:0/2364549495 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7e8102100 con 0x7fd7e806bdf0 2024-09-06T14:23:02.575 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:02.562+0000 7fd7ed544640 1 --2- 172.21.15.162:0/2364549495 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd7e81084a0 0x7fd7e8101840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-06T14:23:02.575 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:02.562+0000 7fd7e7fff640 1 --2- 172.21.15.162:0/2364549495 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd7e81058f0 0x7fd7e8101300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-06T14:23:02.575 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:02.563+0000 7fd7ecd43640 1 --2- 172.21.15.162:0/2364549495 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd7e806bdf0 0x7fd7e8102a90 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fd7dc005ab0 tx=0x7fd7dc032c70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:23:02.576 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:02.563+0000 7fd7e5ffb640 1 -- 172.21.15.162:0/2364549495 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7dc03f030 con 0x7fd7e806bdf0 2024-09-06T14:23:02.576 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:02.563+0000 7fd7eefa7640 1 -- 172.21.15.162:0/2364549495 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7e81a23b0 con 0x7fd7e806bdf0 2024-09-06T14:23:02.576 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:02.563+0000 7fd7e5ffb640 1 -- 172.21.15.162:0/2364549495 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7fd7dc005480 con 0x7fd7e806bdf0 2024-09-06T14:23:02.576 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:02.563+0000 7fd7e5ffb640 1 -- 172.21.15.162:0/2364549495 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7dc035070 con 0x7fd7e806bdf0 2024-09-06T14:23:02.576 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:02.563+0000 7fd7eefa7640 1 -- 172.21.15.162:0/2364549495 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7e81a2860 con 0x7fd7e806bdf0 2024-09-06T14:23:02.576 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:02.564+0000 7fd7eefa7640 1 -- 172.21.15.162:0/2364549495 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7e8105d50 con 0x7fd7e806bdf0 2024-09-06T14:23:02.576 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:02.565+0000 7fd7e5ffb640 1 -- 172.21.15.162:0/2364549495 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 15) ==== 99786+0+0 (secure 0 0 0) 0x7fd7dc042030 con 0x7fd7e806bdf0 2024-09-06T14:23:02.576 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:02.565+0000 7fd7e5ffb640 1 --2- 172.21.15.162:0/2364549495 >> v2:172.21.15.142:6800/853729507 conn(0x7fd7d8077530 0x7fd7d80799f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:23:02.576 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:02.565+0000 7fd7e5ffb640 1 -- 172.21.15.162:0/2364549495 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 3119+0+0 (secure 0 0 0) 0x7fd7dc0b92b0 con 0x7fd7e806bdf0 2024-09-06T14:23:02.576 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:02.565+0000 7fd7e7fff640 1 --2- 172.21.15.162:0/2364549495 >> v2:172.21.15.142:6800/853729507 conn(0x7fd7d8077530 0x7fd7d80799f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:23:02.576 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:02.566+0000 7fd7e7fff640 1 --2- 172.21.15.162:0/2364549495 >> v2:172.21.15.142:6800/853729507 conn(0x7fd7d8077530 0x7fd7d80799f0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd7d40058c0 tx=0x7fd7d4005850 comp rx=0 tx=0).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:23:02.576 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:02.569+0000 7fd7e5ffb640 1 -- 172.21.15.162:0/2364549495 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7fd7dc082f40 con 0x7fd7e806bdf0 2024-09-06T14:23:02.725 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:02.728+0000 7fd7eefa7640 1 -- 172.21.15.162:0/2364549495 --> v2:172.21.15.142:6800/853729507 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fd7e80632f0 con 0x7fd7d8077530 2024-09-06T14:23:03.796 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:03 smithi162 ceph-mon[33787]: osdmap e32: 5 total, 5 up, 5 in 2024-09-06T14:23:03.797 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:03 smithi162 ceph-mon[33787]: from='client.24214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-06T14:23:03.797 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:03 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-06T14:23:03.797 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:03 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-06T14:23:03.797 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:03 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:23:03.797 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:03 smithi162 ceph-mon[33787]: pgmap v114: 1 pgs: 1 remapped+peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-06T14:23:03.853 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:03 smithi142 ceph-mon[29958]: osdmap e32: 5 total, 5 up, 5 in 2024-09-06T14:23:03.854 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:03 smithi142 ceph-mon[29958]: from='client.24214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-06T14:23:03.854 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:03 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-06T14:23:03.854 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:03 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-06T14:23:03.854 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:03 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:23:03.854 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:03 smithi142 ceph-mon[29958]: pgmap v114: 1 pgs: 1 remapped+peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-06T14:23:03.855 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:03 smithi142 ceph-mon[37372]: osdmap e32: 5 total, 5 up, 5 in 2024-09-06T14:23:03.855 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:03 smithi142 ceph-mon[37372]: from='client.24214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-06T14:23:03.855 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:03 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-06T14:23:03.855 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:03 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-06T14:23:03.855 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:03 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:23:03.855 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:03 smithi142 ceph-mon[37372]: pgmap v114: 1 pgs: 1 remapped+peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-06T14:23:06.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:05 smithi142 ceph-mon[29958]: pgmap v115: 1 pgs: 1 remapped+peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-06T14:23:06.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:05 smithi142 ceph-mon[29958]: from='client.? 172.21.15.162:0/3862900803' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "9215d7b7-c268-44bb-b688-74ab1576d1c2"} : dispatch 2024-09-06T14:23:06.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:05 smithi142 ceph-mon[29958]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "9215d7b7-c268-44bb-b688-74ab1576d1c2"} : dispatch 2024-09-06T14:23:06.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:05 smithi142 ceph-mon[29958]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9215d7b7-c268-44bb-b688-74ab1576d1c2"}]': finished 2024-09-06T14:23:06.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:05 smithi142 ceph-mon[29958]: osdmap e33: 6 total, 5 up, 6 in 2024-09-06T14:23:06.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:05 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-06T14:23:06.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:05 smithi142 ceph-mon[29958]: from='client.? 172.21.15.162:0/1065777560' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-06T14:23:06.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:05 smithi142 ceph-mon[37372]: pgmap v115: 1 pgs: 1 remapped+peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-06T14:23:06.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:05 smithi142 ceph-mon[37372]: from='client.? 172.21.15.162:0/3862900803' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "9215d7b7-c268-44bb-b688-74ab1576d1c2"} : dispatch 2024-09-06T14:23:06.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:05 smithi142 ceph-mon[37372]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "9215d7b7-c268-44bb-b688-74ab1576d1c2"} : dispatch 2024-09-06T14:23:06.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:05 smithi142 ceph-mon[37372]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9215d7b7-c268-44bb-b688-74ab1576d1c2"}]': finished 2024-09-06T14:23:06.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:05 smithi142 ceph-mon[37372]: osdmap e33: 6 total, 5 up, 6 in 2024-09-06T14:23:06.210 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:05 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-06T14:23:06.210 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:05 smithi142 ceph-mon[37372]: from='client.? 172.21.15.162:0/1065777560' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-06T14:23:06.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:05 smithi162 ceph-mon[33787]: pgmap v115: 1 pgs: 1 remapped+peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-06T14:23:06.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:05 smithi162 ceph-mon[33787]: from='client.? 172.21.15.162:0/3862900803' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "9215d7b7-c268-44bb-b688-74ab1576d1c2"} : dispatch 2024-09-06T14:23:06.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:05 smithi162 ceph-mon[33787]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "9215d7b7-c268-44bb-b688-74ab1576d1c2"} : dispatch 2024-09-06T14:23:06.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:05 smithi162 ceph-mon[33787]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9215d7b7-c268-44bb-b688-74ab1576d1c2"}]': finished 2024-09-06T14:23:06.298 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:05 smithi162 ceph-mon[33787]: osdmap e33: 6 total, 5 up, 6 in 2024-09-06T14:23:06.298 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:05 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-06T14:23:06.298 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:05 smithi162 ceph-mon[33787]: from='client.? 172.21.15.162:0/1065777560' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-06T14:23:08.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:07 smithi142 ceph-mon[29958]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-06T14:23:08.208 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:07 smithi142 ceph-mon[37372]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-06T14:23:08.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:07 smithi162 ceph-mon[33787]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-06T14:23:10.046 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:09 smithi162 ceph-mon[33787]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-09-06T14:23:10.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:09 smithi142 ceph-mon[29958]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-09-06T14:23:10.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:09 smithi142 ceph-mon[37372]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-09-06T14:23:12.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:11 smithi142 ceph-mon[29958]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-09-06T14:23:12.208 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:11 smithi142 ceph-mon[37372]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-09-06T14:23:12.250 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:11 smithi162 ceph-mon[33787]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-09-06T14:23:13.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:12 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-06T14:23:13.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:12 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:23:13.208 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:12 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-06T14:23:13.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:12 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:23:13.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:12 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-06T14:23:13.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:12 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:23:14.191 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:13 smithi162 ceph-mon[33787]: Deploying daemon osd.5 on smithi162 2024-09-06T14:23:14.191 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:13 smithi162 ceph-mon[33787]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-06T14:23:14.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:13 smithi142 ceph-mon[29958]: Deploying daemon osd.5 on smithi162 2024-09-06T14:23:14.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:13 smithi142 ceph-mon[29958]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-06T14:23:14.208 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:13 smithi142 ceph-mon[37372]: Deploying daemon osd.5 on smithi162 2024-09-06T14:23:14.208 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:13 smithi142 ceph-mon[37372]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-06T14:23:16.046 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:15 smithi162 ceph-mon[33787]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-06T14:23:16.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:15 smithi142 ceph-mon[29958]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-06T14:23:16.208 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:15 smithi142 ceph-mon[37372]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-06T14:23:18.046 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:17 smithi162 ceph-mon[33787]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-09-06T14:23:18.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:17 smithi142 ceph-mon[29958]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-09-06T14:23:18.208 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:17 smithi142 ceph-mon[37372]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-09-06T14:23:19.046 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:18 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-06T14:23:19.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:18 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:23:19.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:18 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:23:19.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:18 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-06T14:23:19.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:18 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:23:19.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:18 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:23:19.208 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:18 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-06T14:23:19.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:18 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:23:19.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:18 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:23:20.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:19 smithi142 ceph-mon[29958]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-06T14:23:20.208 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:19 smithi142 ceph-mon[37372]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-06T14:23:20.223 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:19 smithi162 ceph-mon[33787]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-06T14:23:22.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:21 smithi142 ceph-mon[29958]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-06T14:23:22.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:21 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:23:22.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:21 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:23:22.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:21 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:23:22.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:21 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-06T14:23:22.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:21 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:23:22.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:21 smithi142 ceph-mon[37372]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-06T14:23:22.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:21 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:23:22.210 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:21 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:23:22.210 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:21 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:23:22.210 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:21 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-06T14:23:22.210 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:21 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:23:22.281 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:21 smithi162 ceph-mon[33787]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-06T14:23:22.281 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:21 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:23:22.281 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:21 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:23:22.281 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:21 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:23:22.282 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:21 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-06T14:23:22.282 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:21 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:23:22.321 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:22.324+0000 7fd7e5ffb640 1 -- 172.21.15.162:0/2364549495 <== mgr.14146 v2:172.21.15.142:6800/853729507 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fd7e80632f0 con 0x7fd7d8077530 2024-09-06T14:23:22.322 INFO:teuthology.orchestra.run.smithi162.stdout:Created osd(s) 5 on host 'smithi162' 2024-09-06T14:23:22.323 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:22.326+0000 7fd7eefa7640 1 -- 172.21.15.162:0/2364549495 >> v2:172.21.15.142:6800/853729507 conn(0x7fd7d8077530 msgr2=0x7fd7d80799f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:23:22.323 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:22.326+0000 7fd7eefa7640 1 --2- 172.21.15.162:0/2364549495 >> v2:172.21.15.142:6800/853729507 conn(0x7fd7d8077530 0x7fd7d80799f0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd7d40058c0 tx=0x7fd7d4005850 comp rx=0 tx=0).stop 2024-09-06T14:23:22.323 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:22.326+0000 7fd7eefa7640 1 -- 172.21.15.162:0/2364549495 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd7e806bdf0 msgr2=0x7fd7e8102a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:23:22.323 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:22.326+0000 7fd7eefa7640 1 --2- 172.21.15.162:0/2364549495 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd7e806bdf0 0x7fd7e8102a90 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fd7dc005ab0 tx=0x7fd7dc032c70 comp rx=0 tx=0).stop 2024-09-06T14:23:22.323 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:22.326+0000 7fd7eefa7640 1 -- 172.21.15.162:0/2364549495 shutdown_connections 2024-09-06T14:23:22.323 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:22.326+0000 7fd7eefa7640 1 --2- 172.21.15.162:0/2364549495 >> v2:172.21.15.142:6800/853729507 conn(0x7fd7d8077530 0x7fd7d80799f0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:23:22.324 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:22.326+0000 7fd7eefa7640 1 --2- 172.21.15.162:0/2364549495 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd7e81084a0 0x7fd7e8101840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:23:22.324 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:22.326+0000 7fd7eefa7640 1 --2- 172.21.15.162:0/2364549495 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd7e81058f0 0x7fd7e8101300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:23:22.324 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:22.326+0000 7fd7eefa7640 1 --2- 172.21.15.162:0/2364549495 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd7e806bdf0 0x7fd7e8102a90 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:23:22.324 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:22.326+0000 7fd7eefa7640 1 -- 172.21.15.162:0/2364549495 >> 172.21.15.162:0/2364549495 conn(0x7fd7e80fcf30 msgr2=0x7fd7e8103b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:23:22.324 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:22.326+0000 7fd7eefa7640 1 -- 172.21.15.162:0/2364549495 shutdown_connections 2024-09-06T14:23:22.324 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:22.326+0000 7fd7eefa7640 1 -- 172.21.15.162:0/2364549495 wait complete. 2024-09-06T14:23:22.920 DEBUG:teuthology.orchestra.run.smithi162:osd.5> sudo journalctl -f -n 0 -u ceph-ce32462e-6c5a-11ef-bce4-c7b262605968@osd.5.service 2024-09-06T14:23:22.922 INFO:tasks.cephadm:Deploying osd.6 on smithi162 with /dev/vg_nvme/lv_2... 2024-09-06T14:23:22.922 DEBUG:teuthology.orchestra.run.smithi162:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- lvm zap /dev/vg_nvme/lv_2 2024-09-06T14:23:23.202 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:22 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-06T14:23:23.202 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:22 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:23:23.202 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:22 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:23:23.202 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:22 smithi162 ceph-mon[33787]: from='osd.5 v2:172.21.15.162:6804/3404139106' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-06T14:23:23.202 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:22 smithi162 ceph-mon[33787]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-06T14:23:23.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:22 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-06T14:23:23.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:22 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:23:23.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:22 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:23:23.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:22 smithi142 ceph-mon[29958]: from='osd.5 v2:172.21.15.162:6804/3404139106' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-06T14:23:23.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:22 smithi142 ceph-mon[29958]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-06T14:23:23.208 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:22 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-06T14:23:23.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:22 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:23:23.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:22 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:23:23.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:22 smithi142 ceph-mon[37372]: from='osd.5 v2:172.21.15.162:6804/3404139106' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-06T14:23:23.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:22 smithi142 ceph-mon[37372]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-06T14:23:23.720 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.b/config 2024-09-06T14:23:24.141 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:23 smithi162 ceph-mon[33787]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-06T14:23:24.141 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:23 smithi162 ceph-mon[33787]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-06T14:23:24.141 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:23 smithi162 ceph-mon[33787]: from='osd.5 v2:172.21.15.162:6804/3404139106' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-06T14:23:24.141 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:23 smithi162 ceph-mon[33787]: osdmap e34: 6 total, 5 up, 6 in 2024-09-06T14:23:24.142 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:23 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-06T14:23:24.142 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:23 smithi162 ceph-mon[33787]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-06T14:23:24.208 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:23 smithi142 ceph-mon[37372]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-06T14:23:24.208 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:23 smithi142 ceph-mon[37372]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-06T14:23:24.208 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:23 smithi142 ceph-mon[37372]: from='osd.5 v2:172.21.15.162:6804/3404139106' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-06T14:23:24.208 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:23 smithi142 ceph-mon[37372]: osdmap e34: 6 total, 5 up, 6 in 2024-09-06T14:23:24.208 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:23 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-06T14:23:24.208 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:23 smithi142 ceph-mon[37372]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-06T14:23:24.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:23 smithi142 ceph-mon[29958]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-06T14:23:24.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:23 smithi142 ceph-mon[29958]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-06T14:23:24.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:23 smithi142 ceph-mon[29958]: from='osd.5 v2:172.21.15.162:6804/3404139106' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-06T14:23:24.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:23 smithi142 ceph-mon[29958]: osdmap e34: 6 total, 5 up, 6 in 2024-09-06T14:23:24.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:23 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-06T14:23:24.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:23 smithi142 ceph-mon[29958]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-06T14:23:25.047 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 06 14:23:24 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-5[44099]: 2024-09-06T14:23:24.763+0000 7fbac917e640 -1 osd.5 0 waiting for initial osdmap 2024-09-06T14:23:25.047 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 06 14:23:24 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-5[44099]: 2024-09-06T14:23:24.767+0000 7fbac0f9d640 -1 osd.5 35 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-06T14:23:25.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:25 smithi162 ceph-mon[33787]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-09-06T14:23:25.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:25 smithi162 ceph-mon[33787]: osdmap e35: 6 total, 5 up, 6 in 2024-09-06T14:23:25.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:25 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-06T14:23:25.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:25 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-06T14:23:25.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:25 smithi162 ceph-mon[33787]: from='osd.5 ' entity='osd.5' 2024-09-06T14:23:25.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:25 smithi162 ceph-mon[33787]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-06T14:23:25.708 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:25 smithi142 ceph-mon[37372]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-09-06T14:23:25.708 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:25 smithi142 ceph-mon[37372]: osdmap e35: 6 total, 5 up, 6 in 2024-09-06T14:23:25.708 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:25 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-06T14:23:25.708 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:25 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-06T14:23:25.708 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:25 smithi142 ceph-mon[37372]: from='osd.5 ' entity='osd.5' 2024-09-06T14:23:25.709 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:25 smithi142 ceph-mon[37372]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-06T14:23:25.709 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:25 smithi142 ceph-mon[29958]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-09-06T14:23:25.709 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:25 smithi142 ceph-mon[29958]: osdmap e35: 6 total, 5 up, 6 in 2024-09-06T14:23:25.709 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:25 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-06T14:23:25.709 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:25 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-06T14:23:25.709 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:25 smithi142 ceph-mon[29958]: from='osd.5 ' entity='osd.5' 2024-09-06T14:23:25.709 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:25 smithi142 ceph-mon[29958]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-06T14:23:26.523 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-06T14:23:26.548 DEBUG:teuthology.orchestra.run.smithi162:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph orch daemon add osd smithi162:vg_nvme/lv_2 2024-09-06T14:23:26.571 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:26 smithi162 ceph-mon[33787]: purged_snaps scrub starts 2024-09-06T14:23:26.571 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:26 smithi162 ceph-mon[33787]: purged_snaps scrub ok 2024-09-06T14:23:26.572 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:26 smithi162 ceph-mon[33787]: osd.5 v2:172.21.15.162:6804/3404139106 boot 2024-09-06T14:23:26.572 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:26 smithi162 ceph-mon[33787]: osdmap e36: 6 total, 6 up, 6 in 2024-09-06T14:23:26.572 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:26 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-06T14:23:26.572 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:26 smithi162 ceph-mon[33787]: osdmap e37: 6 total, 6 up, 6 in 2024-09-06T14:23:26.708 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:26 smithi142 ceph-mon[29958]: purged_snaps scrub starts 2024-09-06T14:23:26.708 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:26 smithi142 ceph-mon[29958]: purged_snaps scrub ok 2024-09-06T14:23:26.708 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:26 smithi142 ceph-mon[29958]: osd.5 v2:172.21.15.162:6804/3404139106 boot 2024-09-06T14:23:26.708 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:26 smithi142 ceph-mon[29958]: osdmap e36: 6 total, 6 up, 6 in 2024-09-06T14:23:26.709 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:26 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-06T14:23:26.709 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:26 smithi142 ceph-mon[29958]: osdmap e37: 6 total, 6 up, 6 in 2024-09-06T14:23:26.709 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:26 smithi142 ceph-mon[37372]: purged_snaps scrub starts 2024-09-06T14:23:26.709 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:26 smithi142 ceph-mon[37372]: purged_snaps scrub ok 2024-09-06T14:23:26.709 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:26 smithi142 ceph-mon[37372]: osd.5 v2:172.21.15.162:6804/3404139106 boot 2024-09-06T14:23:26.709 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:26 smithi142 ceph-mon[37372]: osdmap e36: 6 total, 6 up, 6 in 2024-09-06T14:23:26.709 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:26 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-06T14:23:26.709 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:26 smithi142 ceph-mon[37372]: osdmap e37: 6 total, 6 up, 6 in 2024-09-06T14:23:27.090 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.b/config 2024-09-06T14:23:27.485 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:27 smithi162 ceph-mon[33787]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-06T14:23:27.485 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:27 smithi162 ceph-mon[33787]: osdmap e38: 6 total, 6 up, 6 in 2024-09-06T14:23:27.594 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:27.576+0000 7f5f061ad640 1 -- 172.21.15.162:0/4245299043 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5f00105940 msgr2=0x7f5f00107d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:23:27.594 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:27.576+0000 7f5f061ad640 1 --2- 172.21.15.162:0/4245299043 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5f00105940 0x7f5f00107d50 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f5ee8009900 tx=0x7f5ee802a410 comp rx=0 tx=0).stop 2024-09-06T14:23:27.594 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:27.576+0000 7f5f061ad640 1 -- 172.21.15.162:0/4245299043 shutdown_connections 2024-09-06T14:23:27.594 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:27.576+0000 7f5f061ad640 1 --2- 172.21.15.162:0/4245299043 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5f00108490 0x7f5f0010a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:23:27.594 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:27.576+0000 7f5f061ad640 1 --2- 172.21.15.162:0/4245299043 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5f00105940 0x7f5f00107d50 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:23:27.594 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:27.576+0000 7f5f061ad640 1 --2- 172.21.15.162:0/4245299043 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5f0006bb90 0x7f5f00105400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:23:27.594 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:27.576+0000 7f5f061ad640 1 -- 172.21.15.162:0/4245299043 >> 172.21.15.162:0/4245299043 conn(0x7f5f000fcf70 msgr2=0x7f5f000ff390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:23:27.594 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:27.576+0000 7f5f061ad640 1 -- 172.21.15.162:0/4245299043 shutdown_connections 2024-09-06T14:23:27.594 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:27.576+0000 7f5f061ad640 1 -- 172.21.15.162:0/4245299043 wait complete. 2024-09-06T14:23:27.594 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:27.577+0000 7f5f061ad640 1 Processor -- start 2024-09-06T14:23:27.594 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:27.577+0000 7f5f061ad640 1 -- start start 2024-09-06T14:23:27.594 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:27.577+0000 7f5f061ad640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5f0006bb90 0x7f5f001981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:23:27.595 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:27.577+0000 7f5f061ad640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5f00105940 0x7f5f00198840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:23:27.595 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:27.577+0000 7f5f061ad640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5f00108490 0x7f5f00199480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:23:27.595 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:27.577+0000 7f5f061ad640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f5f0010da70 con 0x7f5f0006bb90 2024-09-06T14:23:27.595 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:27.577+0000 7f5f061ad640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5f0010d8f0 con 0x7f5f00108490 2024-09-06T14:23:27.595 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:27.577+0000 7f5f061ad640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7f5f0010dbf0 con 0x7f5f00105940 2024-09-06T14:23:27.595 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:27.577+0000 7f5efffff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5f00108490 0x7f5f00199480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:23:27.595 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:27.577+0000 7f5efffff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5f00108490 0x7f5f00199480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:38026/0 (socket says 172.21.15.162:38026) 2024-09-06T14:23:27.595 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:27.577+0000 7f5efffff640 1 -- 172.21.15.162:0/1908302676 learned_addr learned my addr 172.21.15.162:0/1908302676 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-09-06T14:23:27.595 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:27.577+0000 7f5eff7fe640 1 --2- 172.21.15.162:0/1908302676 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5f0006bb90 0x7f5f001981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:23:27.595 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:27.577+0000 7f5efeffd640 1 --2- 172.21.15.162:0/1908302676 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5f00105940 0x7f5f00198840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:23:27.595 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:27.578+0000 7f5efffff640 1 -- 172.21.15.162:0/1908302676 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5f00105940 msgr2=0x7f5f00198840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:23:27.595 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:27.578+0000 7f5efffff640 1 --2- 172.21.15.162:0/1908302676 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5f00105940 0x7f5f00198840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:23:27.595 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:27.578+0000 7f5efffff640 1 -- 172.21.15.162:0/1908302676 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5f0006bb90 msgr2=0x7f5f001981f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:23:27.595 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:27.578+0000 7f5efffff640 1 --2- 172.21.15.162:0/1908302676 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5f0006bb90 0x7f5f001981f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:23:27.596 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:27.578+0000 7f5efffff640 1 -- 172.21.15.162:0/1908302676 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f00199d20 con 0x7f5f00108490 2024-09-06T14:23:27.596 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:27.578+0000 7f5efffff640 1 --2- 172.21.15.162:0/1908302676 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5f00108490 0x7f5f00199480 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f5ef800eaa0 tx=0x7f5ef800ef60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:23:27.596 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:27.578+0000 7f5efcff9640 1 -- 172.21.15.162:0/1908302676 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ef800cdb0 con 0x7f5f00108490 2024-09-06T14:23:27.596 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:27.578+0000 7f5efcff9640 1 -- 172.21.15.162:0/1908302676 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7f5ef8005020 con 0x7f5f00108490 2024-09-06T14:23:27.596 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:27.578+0000 7f5f061ad640 1 -- 172.21.15.162:0/1908302676 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f0019a010 con 0x7f5f00108490 2024-09-06T14:23:27.596 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:27.578+0000 7f5efcff9640 1 -- 172.21.15.162:0/1908302676 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ef80116d0 con 0x7f5f00108490 2024-09-06T14:23:27.596 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:27.578+0000 7f5f061ad640 1 -- 172.21.15.162:0/1908302676 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f001a06f0 con 0x7f5f00108490 2024-09-06T14:23:27.596 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:27.579+0000 7f5f061ad640 1 -- 172.21.15.162:0/1908302676 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f00105d40 con 0x7f5f00108490 2024-09-06T14:23:27.596 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:27.580+0000 7f5efcff9640 1 -- 172.21.15.162:0/1908302676 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 15) ==== 99786+0+0 (secure 0 0 0) 0x7f5ef8004a40 con 0x7f5f00108490 2024-09-06T14:23:27.596 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:27.581+0000 7f5efcff9640 1 --2- 172.21.15.162:0/1908302676 >> v2:172.21.15.142:6800/853729507 conn(0x7f5eec077530 0x7f5eec0799f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:23:27.596 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:27.581+0000 7f5efcff9640 1 -- 172.21.15.162:0/1908302676 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 3411+0+0 (secure 0 0 0) 0x7f5ef8098e30 con 0x7f5f00108490 2024-09-06T14:23:27.596 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:27.581+0000 7f5eff7fe640 1 --2- 172.21.15.162:0/1908302676 >> v2:172.21.15.142:6800/853729507 conn(0x7f5eec077530 0x7f5eec0799f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:23:27.596 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:27.582+0000 7f5eff7fe640 1 --2- 172.21.15.162:0/1908302676 >> v2:172.21.15.142:6800/853729507 conn(0x7f5eec077530 0x7f5eec0799f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f5ef0005d00 tx=0x7f5ef000b410 comp rx=0 tx=0).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:23:27.596 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:27.585+0000 7f5efcff9640 1 -- 172.21.15.162:0/1908302676 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7f5ef8016030 con 0x7f5f00108490 2024-09-06T14:23:27.708 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:27 smithi142 ceph-mon[29958]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-06T14:23:27.708 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:27 smithi142 ceph-mon[29958]: osdmap e38: 6 total, 6 up, 6 in 2024-09-06T14:23:27.708 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:27 smithi142 ceph-mon[37372]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-06T14:23:27.709 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:27 smithi142 ceph-mon[37372]: osdmap e38: 6 total, 6 up, 6 in 2024-09-06T14:23:27.743 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:27.747+0000 7f5f061ad640 1 -- 172.21.15.162:0/1908302676 --> v2:172.21.15.142:6800/853729507 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f5f0010a030 con 0x7f5eec077530 2024-09-06T14:23:28.398 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:28 smithi162 ceph-mon[33787]: from='client.24241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-06T14:23:28.398 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:28 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-06T14:23:28.398 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:28 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-06T14:23:28.398 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:28 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:23:28.398 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:28 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:23:28.398 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:28 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:23:28.399 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:28 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-06T14:23:28.399 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:28 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-06T14:23:28.399 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:28 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:23:28.399 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:28 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:23:28.399 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:28 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-06T14:23:28.399 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:28 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:23:28.708 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:28 smithi142 ceph-mon[29958]: from='client.24241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-06T14:23:28.708 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:28 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-06T14:23:28.709 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:28 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-06T14:23:28.709 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:28 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:23:28.709 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:28 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:23:28.709 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:28 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:23:28.709 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:28 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-06T14:23:28.709 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:28 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-06T14:23:28.709 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:28 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:23:28.709 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:28 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:23:28.709 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:28 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-06T14:23:28.709 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:28 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:23:28.709 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:28 smithi142 ceph-mon[37372]: from='client.24241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-06T14:23:28.709 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:28 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-06T14:23:28.710 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:28 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-06T14:23:28.710 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:28 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:23:28.710 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:28 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:23:28.710 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:28 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:23:28.710 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:28 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-06T14:23:28.710 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:28 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-06T14:23:28.710 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:28 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:23:28.710 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:28 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:23:28.710 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:28 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-06T14:23:28.710 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:28 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:23:29.582 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:29 smithi162 ceph-mon[33787]: Detected new or changed devices on smithi162 2024-09-06T14:23:29.582 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:29 smithi162 ceph-mon[33787]: Adjusting osd_memory_target on smithi162 to 8544M 2024-09-06T14:23:29.582 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:29 smithi162 ceph-mon[33787]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-06T14:23:29.708 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:29 smithi142 ceph-mon[37372]: Detected new or changed devices on smithi162 2024-09-06T14:23:29.708 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:29 smithi142 ceph-mon[37372]: Adjusting osd_memory_target on smithi162 to 8544M 2024-09-06T14:23:29.708 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:29 smithi142 ceph-mon[37372]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-06T14:23:29.708 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:29 smithi142 ceph-mon[29958]: Detected new or changed devices on smithi162 2024-09-06T14:23:29.709 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:29 smithi142 ceph-mon[29958]: Adjusting osd_memory_target on smithi162 to 8544M 2024-09-06T14:23:29.709 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:29 smithi142 ceph-mon[29958]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-06T14:23:30.708 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:30 smithi142 ceph-mon[29958]: from='client.? 172.21.15.162:0/3111711552' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "457940ad-3624-4613-98ce-817baa5f2d1d"} : dispatch 2024-09-06T14:23:30.708 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:30 smithi142 ceph-mon[29958]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "457940ad-3624-4613-98ce-817baa5f2d1d"} : dispatch 2024-09-06T14:23:30.709 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:30 smithi142 ceph-mon[29958]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "457940ad-3624-4613-98ce-817baa5f2d1d"}]': finished 2024-09-06T14:23:30.709 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:30 smithi142 ceph-mon[29958]: osdmap e39: 7 total, 6 up, 7 in 2024-09-06T14:23:30.709 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:30 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-06T14:23:30.709 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:30 smithi142 ceph-mon[37372]: from='client.? 172.21.15.162:0/3111711552' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "457940ad-3624-4613-98ce-817baa5f2d1d"} : dispatch 2024-09-06T14:23:30.709 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:30 smithi142 ceph-mon[37372]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "457940ad-3624-4613-98ce-817baa5f2d1d"} : dispatch 2024-09-06T14:23:30.709 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:30 smithi142 ceph-mon[37372]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "457940ad-3624-4613-98ce-817baa5f2d1d"}]': finished 2024-09-06T14:23:30.709 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:30 smithi142 ceph-mon[37372]: osdmap e39: 7 total, 6 up, 7 in 2024-09-06T14:23:30.709 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:30 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-06T14:23:30.796 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:30 smithi162 ceph-mon[33787]: from='client.? 172.21.15.162:0/3111711552' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "457940ad-3624-4613-98ce-817baa5f2d1d"} : dispatch 2024-09-06T14:23:30.796 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:30 smithi162 ceph-mon[33787]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "457940ad-3624-4613-98ce-817baa5f2d1d"} : dispatch 2024-09-06T14:23:30.797 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:30 smithi162 ceph-mon[33787]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "457940ad-3624-4613-98ce-817baa5f2d1d"}]': finished 2024-09-06T14:23:30.797 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:30 smithi162 ceph-mon[33787]: osdmap e39: 7 total, 6 up, 7 in 2024-09-06T14:23:30.797 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:30 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-06T14:23:31.708 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:31 smithi142 ceph-mon[29958]: from='client.? 172.21.15.162:0/2215922477' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-06T14:23:31.708 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:31 smithi142 ceph-mon[29958]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-06T14:23:31.708 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:31 smithi142 ceph-mon[37372]: from='client.? 172.21.15.162:0/2215922477' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-06T14:23:31.709 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:31 smithi142 ceph-mon[37372]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-06T14:23:31.796 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:31 smithi162 ceph-mon[33787]: from='client.? 172.21.15.162:0/2215922477' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-06T14:23:31.796 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:31 smithi162 ceph-mon[33787]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-06T14:23:34.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:33 smithi142 ceph-mon[29958]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-06T14:23:34.208 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:33 smithi142 ceph-mon[37372]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-06T14:23:34.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:33 smithi162 ceph-mon[33787]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-06T14:23:36.194 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:35 smithi162 ceph-mon[33787]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-06T14:23:36.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:35 smithi142 ceph-mon[29958]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-06T14:23:36.208 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:35 smithi142 ceph-mon[37372]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-06T14:23:38.046 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:37 smithi162 ceph-mon[33787]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-06T14:23:38.046 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:37 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-06T14:23:38.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:37 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:23:38.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:37 smithi142 ceph-mon[29958]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-06T14:23:38.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:37 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-06T14:23:38.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:37 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:23:38.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:37 smithi142 ceph-mon[37372]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-06T14:23:38.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:37 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-06T14:23:38.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:37 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:23:39.098 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:38 smithi162 ceph-mon[33787]: Deploying daemon osd.6 on smithi162 2024-09-06T14:23:39.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:38 smithi142 ceph-mon[29958]: Deploying daemon osd.6 on smithi162 2024-09-06T14:23:39.208 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:38 smithi142 ceph-mon[37372]: Deploying daemon osd.6 on smithi162 2024-09-06T14:23:40.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:39 smithi142 ceph-mon[29958]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-06T14:23:40.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:39 smithi142 ceph-mon[37372]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-06T14:23:40.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:39 smithi162 ceph-mon[33787]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-06T14:23:42.208 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:41 smithi142 ceph-mon[37372]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-06T14:23:42.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:41 smithi142 ceph-mon[29958]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-06T14:23:42.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:41 smithi162 ceph-mon[33787]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-06T14:23:44.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:43 smithi142 ceph-mon[29958]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-06T14:23:44.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:43 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-06T14:23:44.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:43 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:23:44.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:43 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:23:44.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:43 smithi142 ceph-mon[37372]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-06T14:23:44.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:43 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-06T14:23:44.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:43 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:23:44.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:43 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:23:44.216 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:43 smithi162 ceph-mon[33787]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-06T14:23:44.216 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:43 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-06T14:23:44.216 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:43 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:23:44.216 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:43 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:23:46.203 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:45 smithi162 ceph-mon[33787]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-06T14:23:46.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:45 smithi142 ceph-mon[29958]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-06T14:23:46.208 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:45 smithi142 ceph-mon[37372]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-06T14:23:47.388 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:47.392+0000 7f5efcff9640 1 -- 172.21.15.162:0/1908302676 <== mgr.14146 v2:172.21.15.142:6800/853729507 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f5f0010a030 con 0x7f5eec077530 2024-09-06T14:23:47.389 INFO:teuthology.orchestra.run.smithi162.stdout:Created osd(s) 6 on host 'smithi162' 2024-09-06T14:23:47.390 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:47.394+0000 7f5f061ad640 1 -- 172.21.15.162:0/1908302676 >> v2:172.21.15.142:6800/853729507 conn(0x7f5eec077530 msgr2=0x7f5eec0799f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:23:47.390 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:47.394+0000 7f5f061ad640 1 --2- 172.21.15.162:0/1908302676 >> v2:172.21.15.142:6800/853729507 conn(0x7f5eec077530 0x7f5eec0799f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f5ef0005d00 tx=0x7f5ef000b410 comp rx=0 tx=0).stop 2024-09-06T14:23:47.391 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:47.394+0000 7f5f061ad640 1 -- 172.21.15.162:0/1908302676 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5f00108490 msgr2=0x7f5f00199480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:23:47.391 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:47.394+0000 7f5f061ad640 1 --2- 172.21.15.162:0/1908302676 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5f00108490 0x7f5f00199480 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f5ef800eaa0 tx=0x7f5ef800ef60 comp rx=0 tx=0).stop 2024-09-06T14:23:47.391 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:47.394+0000 7f5f061ad640 1 -- 172.21.15.162:0/1908302676 shutdown_connections 2024-09-06T14:23:47.391 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:47.394+0000 7f5f061ad640 1 --2- 172.21.15.162:0/1908302676 >> v2:172.21.15.142:6800/853729507 conn(0x7f5eec077530 0x7f5eec0799f0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:23:47.391 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:47.394+0000 7f5f061ad640 1 --2- 172.21.15.162:0/1908302676 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5f00108490 0x7f5f00199480 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:23:47.391 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:47.394+0000 7f5f061ad640 1 --2- 172.21.15.162:0/1908302676 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5f00105940 0x7f5f00198840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:23:47.391 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:47.394+0000 7f5f061ad640 1 --2- 172.21.15.162:0/1908302676 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5f0006bb90 0x7f5f001981f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:23:47.391 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:47.394+0000 7f5f061ad640 1 -- 172.21.15.162:0/1908302676 >> 172.21.15.162:0/1908302676 conn(0x7f5f000fcf70 msgr2=0x7f5f00108910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:23:47.391 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:47.394+0000 7f5f061ad640 1 -- 172.21.15.162:0/1908302676 shutdown_connections 2024-09-06T14:23:47.392 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:47.394+0000 7f5f061ad640 1 -- 172.21.15.162:0/1908302676 wait complete. 2024-09-06T14:23:47.708 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:47 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:23:47.709 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:47 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:23:47.709 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:47 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:23:47.709 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:47 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-06T14:23:47.709 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:47 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:23:47.709 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:47 smithi142 ceph-mon[29958]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-06T14:23:47.709 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:47 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-06T14:23:47.709 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:47 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:23:47.709 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:47 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:23:47.709 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:47 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:23:47.709 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:47 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:23:47.710 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:47 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:23:47.710 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:47 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-06T14:23:47.710 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:47 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:23:47.710 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:47 smithi142 ceph-mon[37372]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-06T14:23:47.710 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:47 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-06T14:23:47.710 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:47 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:23:47.710 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:47 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:23:47.743 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:47 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:23:47.743 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:47 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:23:47.743 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:47 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:23:47.743 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:47 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-06T14:23:47.743 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:47 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:23:47.743 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:47 smithi162 ceph-mon[33787]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-06T14:23:47.743 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:47 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-06T14:23:47.743 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:47 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:23:47.743 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:47 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:23:48.080 DEBUG:teuthology.orchestra.run.smithi162:osd.6> sudo journalctl -f -n 0 -u ceph-ce32462e-6c5a-11ef-bce4-c7b262605968@osd.6.service 2024-09-06T14:23:48.123 INFO:tasks.cephadm:Deploying osd.7 on smithi162 with /dev/vg_nvme/lv_1... 2024-09-06T14:23:48.123 DEBUG:teuthology.orchestra.run.smithi162:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- lvm zap /dev/vg_nvme/lv_1 2024-09-06T14:23:48.796 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:48 smithi162 ceph-mon[33787]: from='osd.6 v2:172.21.15.162:6808/1155174053' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-06T14:23:48.797 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:48 smithi162 ceph-mon[33787]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-06T14:23:48.797 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 06 14:23:48 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-6[49599]: 2024-09-06T14:23:48.461+0000 7fef324b7640 -1 osd.6 0 log_to_monitors true 2024-09-06T14:23:48.958 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:48 smithi142 ceph-mon[29958]: from='osd.6 v2:172.21.15.162:6808/1155174053' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-06T14:23:48.958 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:48 smithi142 ceph-mon[29958]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-06T14:23:48.959 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:48 smithi142 ceph-mon[37372]: from='osd.6 v2:172.21.15.162:6808/1155174053' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-06T14:23:48.959 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:48 smithi142 ceph-mon[37372]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-06T14:23:49.271 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.b/config 2024-09-06T14:23:49.743 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 06 14:23:49 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-6[49599]: 2024-09-06T14:23:49.531+0000 7fef2e43a640 -1 osd.6 0 waiting for initial osdmap 2024-09-06T14:23:49.743 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 06 14:23:49 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-6[49599]: 2024-09-06T14:23:49.535+0000 7fef2a261640 -1 osd.6 41 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-06T14:23:49.743 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:49 smithi162 ceph-mon[33787]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-06T14:23:49.744 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:49 smithi162 ceph-mon[33787]: from='osd.6 v2:172.21.15.162:6808/1155174053' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-06T14:23:49.744 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:49 smithi162 ceph-mon[33787]: osdmap e40: 7 total, 6 up, 7 in 2024-09-06T14:23:49.744 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:49 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-06T14:23:49.744 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:49 smithi162 ceph-mon[33787]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-06T14:23:49.744 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:49 smithi162 ceph-mon[33787]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-06T14:23:49.958 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:49 smithi142 ceph-mon[29958]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-06T14:23:49.958 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:49 smithi142 ceph-mon[29958]: from='osd.6 v2:172.21.15.162:6808/1155174053' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-06T14:23:49.958 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:49 smithi142 ceph-mon[29958]: osdmap e40: 7 total, 6 up, 7 in 2024-09-06T14:23:49.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:49 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-06T14:23:49.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:49 smithi142 ceph-mon[29958]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-06T14:23:49.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:49 smithi142 ceph-mon[29958]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-06T14:23:49.959 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:49 smithi142 ceph-mon[37372]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-06T14:23:49.959 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:49 smithi142 ceph-mon[37372]: from='osd.6 v2:172.21.15.162:6808/1155174053' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-06T14:23:49.959 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:49 smithi142 ceph-mon[37372]: osdmap e40: 7 total, 6 up, 7 in 2024-09-06T14:23:49.959 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:49 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-06T14:23:49.959 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:49 smithi142 ceph-mon[37372]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-06T14:23:49.959 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:49 smithi142 ceph-mon[37372]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-06T14:23:50.658 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:50 smithi162 ceph-mon[33787]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-09-06T14:23:50.658 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:50 smithi162 ceph-mon[33787]: osdmap e41: 7 total, 6 up, 7 in 2024-09-06T14:23:50.659 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:50 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-06T14:23:50.659 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:50 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-06T14:23:50.958 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:50 smithi142 ceph-mon[29958]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-09-06T14:23:50.958 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:50 smithi142 ceph-mon[29958]: osdmap e41: 7 total, 6 up, 7 in 2024-09-06T14:23:50.958 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:50 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-06T14:23:50.958 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:50 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-06T14:23:50.959 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:50 smithi142 ceph-mon[37372]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-09-06T14:23:50.959 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:50 smithi142 ceph-mon[37372]: osdmap e41: 7 total, 6 up, 7 in 2024-09-06T14:23:50.959 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:50 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-06T14:23:50.959 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:50 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-06T14:23:51.789 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:51 smithi162 ceph-mon[33787]: purged_snaps scrub starts 2024-09-06T14:23:51.789 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:51 smithi162 ceph-mon[33787]: purged_snaps scrub ok 2024-09-06T14:23:51.789 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:51 smithi162 ceph-mon[33787]: osd.6 v2:172.21.15.162:6808/1155174053 boot 2024-09-06T14:23:51.789 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:51 smithi162 ceph-mon[33787]: osdmap e42: 7 total, 7 up, 7 in 2024-09-06T14:23:51.789 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:51 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-06T14:23:51.790 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:51 smithi162 ceph-mon[33787]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-06T14:23:51.958 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:51 smithi142 ceph-mon[29958]: purged_snaps scrub starts 2024-09-06T14:23:51.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:51 smithi142 ceph-mon[29958]: purged_snaps scrub ok 2024-09-06T14:23:51.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:51 smithi142 ceph-mon[29958]: osd.6 v2:172.21.15.162:6808/1155174053 boot 2024-09-06T14:23:51.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:51 smithi142 ceph-mon[29958]: osdmap e42: 7 total, 7 up, 7 in 2024-09-06T14:23:51.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:51 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-06T14:23:51.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:51 smithi142 ceph-mon[29958]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-06T14:23:51.960 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:51 smithi142 ceph-mon[37372]: purged_snaps scrub starts 2024-09-06T14:23:51.960 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:51 smithi142 ceph-mon[37372]: purged_snaps scrub ok 2024-09-06T14:23:51.960 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:51 smithi142 ceph-mon[37372]: osd.6 v2:172.21.15.162:6808/1155174053 boot 2024-09-06T14:23:51.960 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:51 smithi142 ceph-mon[37372]: osdmap e42: 7 total, 7 up, 7 in 2024-09-06T14:23:51.960 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:51 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-06T14:23:51.960 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:51 smithi142 ceph-mon[37372]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-06T14:23:52.793 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:52 smithi162 ceph-mon[33787]: osdmap e43: 7 total, 7 up, 7 in 2024-09-06T14:23:52.793 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:52 smithi162 ceph-mon[33787]: Detected new or changed devices on smithi162 2024-09-06T14:23:52.794 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:52 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:23:52.794 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:52 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:23:52.794 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:52 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-06T14:23:52.794 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:52 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-06T14:23:52.794 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:52 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-06T14:23:52.794 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:52 smithi162 ceph-mon[33787]: Adjusting osd_memory_target on smithi162 to 5696M 2024-09-06T14:23:52.795 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:52 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:23:52.795 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:52 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:23:52.795 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:52 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-06T14:23:52.795 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:52 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:23:52.958 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:52 smithi142 ceph-mon[29958]: osdmap e43: 7 total, 7 up, 7 in 2024-09-06T14:23:52.958 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:52 smithi142 ceph-mon[29958]: Detected new or changed devices on smithi162 2024-09-06T14:23:52.958 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:52 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:23:52.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:52 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:23:52.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:52 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-06T14:23:52.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:52 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-06T14:23:52.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:52 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-06T14:23:52.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:52 smithi142 ceph-mon[29958]: Adjusting osd_memory_target on smithi162 to 5696M 2024-09-06T14:23:52.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:52 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:23:52.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:52 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:23:52.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:52 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-06T14:23:52.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:52 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:23:52.959 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:52 smithi142 ceph-mon[37372]: osdmap e43: 7 total, 7 up, 7 in 2024-09-06T14:23:52.959 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:52 smithi142 ceph-mon[37372]: Detected new or changed devices on smithi162 2024-09-06T14:23:52.960 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:52 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:23:52.960 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:52 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:23:52.960 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:52 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-06T14:23:52.960 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:52 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-06T14:23:52.960 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:52 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-06T14:23:52.960 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:52 smithi142 ceph-mon[37372]: Adjusting osd_memory_target on smithi162 to 5696M 2024-09-06T14:23:52.960 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:52 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:23:52.960 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:52 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:23:52.960 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:52 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-06T14:23:52.960 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:52 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:23:52.991 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-06T14:23:53.017 DEBUG:teuthology.orchestra.run.smithi162:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph orch daemon add osd smithi162:vg_nvme/lv_1 2024-09-06T14:23:53.267 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.b/config 2024-09-06T14:23:53.695 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:53 smithi162 ceph-mon[33787]: osdmap e44: 7 total, 7 up, 7 in 2024-09-06T14:23:53.695 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:53 smithi162 ceph-mon[33787]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-06T14:23:53.740 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:53.715+0000 7f2d6f70d640 1 -- 172.21.15.162:0/819975901 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2d681036d0 msgr2=0x7f2d68105b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:23:53.740 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:53.715+0000 7f2d6f70d640 1 --2- 172.21.15.162:0/819975901 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2d681036d0 0x7f2d68105b60 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f2d58009900 tx=0x7f2d5802a3f0 comp rx=0 tx=0).stop 2024-09-06T14:23:53.741 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:53.715+0000 7f2d6f70d640 1 -- 172.21.15.162:0/819975901 shutdown_connections 2024-09-06T14:23:53.741 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:53.715+0000 7f2d6f70d640 1 --2- 172.21.15.162:0/819975901 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2d68106280 0x7f2d68106730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:23:53.741 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:53.715+0000 7f2d6f70d640 1 --2- 172.21.15.162:0/819975901 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2d681036d0 0x7f2d68105b60 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:23:53.741 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:53.715+0000 7f2d6f70d640 1 --2- 172.21.15.162:0/819975901 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2d6806be10 0x7f2d68103140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:23:53.741 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:53.715+0000 7f2d6f70d640 1 -- 172.21.15.162:0/819975901 >> 172.21.15.162:0/819975901 conn(0x7f2d680fcf20 msgr2=0x7f2d680ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:23:53.741 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:53.715+0000 7f2d6f70d640 1 -- 172.21.15.162:0/819975901 shutdown_connections 2024-09-06T14:23:53.741 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:53.715+0000 7f2d6f70d640 1 -- 172.21.15.162:0/819975901 wait complete. 2024-09-06T14:23:53.741 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:53.716+0000 7f2d6f70d640 1 Processor -- start 2024-09-06T14:23:53.741 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:53.716+0000 7f2d6f70d640 1 -- start start 2024-09-06T14:23:53.741 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:53.716+0000 7f2d6f70d640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2d6806be10 0x7f2d6819e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:23:53.741 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:53.716+0000 7f2d6f70d640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2d681036d0 0x7f2d6819ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:23:53.741 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:53.716+0000 7f2d6f70d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2d68106280 0x7f2d6819f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:23:53.741 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:53.716+0000 7f2d6f70d640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f2d68114070 con 0x7f2d681036d0 2024-09-06T14:23:53.742 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:53.716+0000 7f2d6f70d640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f2d68113ef0 con 0x7f2d68106280 2024-09-06T14:23:53.742 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:53.716+0000 7f2d6f70d640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7f2d681141f0 con 0x7f2d6806be10 2024-09-06T14:23:53.742 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:53.716+0000 7f2d6dcaa640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2d68106280 0x7f2d6819f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:23:53.742 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:53.716+0000 7f2d6dcaa640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2d68106280 0x7f2d6819f9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:56352/0 (socket says 172.21.15.162:56352) 2024-09-06T14:23:53.742 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:53.716+0000 7f2d6dcaa640 1 -- 172.21.15.162:0/620178586 learned_addr learned my addr 172.21.15.162:0/620178586 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-09-06T14:23:53.742 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:53.716+0000 7f2d6d4a9640 1 --2- 172.21.15.162:0/620178586 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2d6806be10 0x7f2d6819e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:23:53.742 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:53.716+0000 7f2d6cca8640 1 --2- 172.21.15.162:0/620178586 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2d681036d0 0x7f2d6819ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:23:53.742 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:53.717+0000 7f2d6dcaa640 1 -- 172.21.15.162:0/620178586 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2d6806be10 msgr2=0x7f2d6819e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:23:53.742 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:53.717+0000 7f2d6dcaa640 1 --2- 172.21.15.162:0/620178586 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2d6806be10 0x7f2d6819e740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:23:53.743 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:53.717+0000 7f2d6dcaa640 1 -- 172.21.15.162:0/620178586 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2d681036d0 msgr2=0x7f2d6819ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:23:53.743 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:53.717+0000 7f2d6dcaa640 1 --2- 172.21.15.162:0/620178586 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2d681036d0 0x7f2d6819ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:23:53.743 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:53.717+0000 7f2d6dcaa640 1 -- 172.21.15.162:0/620178586 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d681a0270 con 0x7f2d68106280 2024-09-06T14:23:53.743 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:53.717+0000 7f2d6dcaa640 1 --2- 172.21.15.162:0/620178586 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2d68106280 0x7f2d6819f9d0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f2d6000cce0 tx=0x7f2d6000a620 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:23:53.743 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:53.717+0000 7f2d5e7fc640 1 -- 172.21.15.162:0/620178586 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d6000f450 con 0x7f2d68106280 2024-09-06T14:23:53.743 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:53.717+0000 7f2d5e7fc640 1 -- 172.21.15.162:0/620178586 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7f2d6000fa70 con 0x7f2d68106280 2024-09-06T14:23:53.743 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:53.717+0000 7f2d6f70d640 1 -- 172.21.15.162:0/620178586 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d681a0560 con 0x7f2d68106280 2024-09-06T14:23:53.743 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:53.717+0000 7f2d5e7fc640 1 -- 172.21.15.162:0/620178586 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d60018930 con 0x7f2d68106280 2024-09-06T14:23:53.744 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:53.717+0000 7f2d6f70d640 1 -- 172.21.15.162:0/620178586 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d681a6c40 con 0x7f2d68106280 2024-09-06T14:23:53.744 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:53.718+0000 7f2d6f70d640 1 -- 172.21.15.162:0/620178586 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d68106730 con 0x7f2d68106280 2024-09-06T14:23:53.744 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:53.719+0000 7f2d5e7fc640 1 -- 172.21.15.162:0/620178586 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 15) ==== 99786+0+0 (secure 0 0 0) 0x7f2d6000f5f0 con 0x7f2d68106280 2024-09-06T14:23:53.744 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:53.719+0000 7f2d5e7fc640 1 --2- 172.21.15.162:0/620178586 >> v2:172.21.15.142:6800/853729507 conn(0x7f2d54077530 0x7f2d540799f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:23:53.744 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:53.719+0000 7f2d5e7fc640 1 -- 172.21.15.162:0/620178586 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 3703+0+0 (secure 0 0 0) 0x7f2d600992a0 con 0x7f2d68106280 2024-09-06T14:23:53.744 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:53.719+0000 7f2d6d4a9640 1 --2- 172.21.15.162:0/620178586 >> v2:172.21.15.142:6800/853729507 conn(0x7f2d54077530 0x7f2d540799f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:23:53.744 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:53.720+0000 7f2d6d4a9640 1 --2- 172.21.15.162:0/620178586 >> v2:172.21.15.142:6800/853729507 conn(0x7f2d54077530 0x7f2d540799f0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2d6400abe0 tx=0x7f2d6400a2d0 comp rx=0 tx=0).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:23:53.744 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:53.723+0000 7f2d5e7fc640 1 -- 172.21.15.162:0/620178586 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7f2d60014030 con 0x7f2d68106280 2024-09-06T14:23:53.882 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:23:53.885+0000 7f2d6f70d640 1 -- 172.21.15.162:0/620178586 --> v2:172.21.15.142:6800/853729507 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f2d6802e5c0 con 0x7f2d54077530 2024-09-06T14:23:53.958 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:53 smithi142 ceph-mon[29958]: osdmap e44: 7 total, 7 up, 7 in 2024-09-06T14:23:53.958 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:53 smithi142 ceph-mon[29958]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-06T14:23:53.959 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:53 smithi142 ceph-mon[37372]: osdmap e44: 7 total, 7 up, 7 in 2024-09-06T14:23:53.959 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:53 smithi142 ceph-mon[37372]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-06T14:23:54.958 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:54 smithi142 ceph-mon[29958]: from='client.24268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-06T14:23:54.958 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:54 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-06T14:23:54.958 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:54 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-06T14:23:54.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:54 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:23:54.959 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:54 smithi142 ceph-mon[37372]: from='client.24268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-06T14:23:54.959 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:54 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-06T14:23:54.959 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:54 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-06T14:23:54.959 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:54 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:23:55.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:54 smithi162 ceph-mon[33787]: from='client.24268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-06T14:23:55.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:54 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-06T14:23:55.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:54 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-06T14:23:55.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:54 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:23:55.958 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:55 smithi142 ceph-mon[29958]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-06T14:23:55.958 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:55 smithi142 ceph-mon[37372]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-06T14:23:55.959 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:55 smithi162 ceph-mon[33787]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-06T14:23:56.797 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:56 smithi162 ceph-mon[33787]: from='client.? 172.21.15.162:0/3799285352' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "b92b364c-849e-4971-be37-6c3f440e7cec"} : dispatch 2024-09-06T14:23:56.797 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:56 smithi162 ceph-mon[33787]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "b92b364c-849e-4971-be37-6c3f440e7cec"} : dispatch 2024-09-06T14:23:56.797 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:56 smithi162 ceph-mon[33787]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b92b364c-849e-4971-be37-6c3f440e7cec"}]': finished 2024-09-06T14:23:56.797 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:56 smithi162 ceph-mon[33787]: osdmap e45: 8 total, 7 up, 8 in 2024-09-06T14:23:56.797 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:56 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-06T14:23:56.958 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:56 smithi142 ceph-mon[29958]: from='client.? 172.21.15.162:0/3799285352' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "b92b364c-849e-4971-be37-6c3f440e7cec"} : dispatch 2024-09-06T14:23:56.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:56 smithi142 ceph-mon[29958]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "b92b364c-849e-4971-be37-6c3f440e7cec"} : dispatch 2024-09-06T14:23:56.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:56 smithi142 ceph-mon[29958]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b92b364c-849e-4971-be37-6c3f440e7cec"}]': finished 2024-09-06T14:23:56.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:56 smithi142 ceph-mon[29958]: osdmap e45: 8 total, 7 up, 8 in 2024-09-06T14:23:56.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:56 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-06T14:23:56.959 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:56 smithi142 ceph-mon[37372]: from='client.? 172.21.15.162:0/3799285352' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "b92b364c-849e-4971-be37-6c3f440e7cec"} : dispatch 2024-09-06T14:23:56.959 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:56 smithi142 ceph-mon[37372]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "b92b364c-849e-4971-be37-6c3f440e7cec"} : dispatch 2024-09-06T14:23:56.959 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:56 smithi142 ceph-mon[37372]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b92b364c-849e-4971-be37-6c3f440e7cec"}]': finished 2024-09-06T14:23:56.960 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:56 smithi142 ceph-mon[37372]: osdmap e45: 8 total, 7 up, 8 in 2024-09-06T14:23:56.960 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:56 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-06T14:23:57.958 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:57 smithi142 ceph-mon[29958]: from='client.? 172.21.15.162:0/3274843226' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-06T14:23:57.958 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:57 smithi142 ceph-mon[29958]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-06T14:23:57.959 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:57 smithi142 ceph-mon[37372]: from='client.? 172.21.15.162:0/3274843226' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-06T14:23:57.959 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:57 smithi142 ceph-mon[37372]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-06T14:23:58.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:57 smithi162 ceph-mon[33787]: from='client.? 172.21.15.162:0/3274843226' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-06T14:23:58.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:57 smithi162 ceph-mon[33787]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-06T14:24:00.208 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:23:59 smithi142 ceph-mon[37372]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-06T14:24:00.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:23:59 smithi142 ceph-mon[29958]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-06T14:24:00.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:23:59 smithi162 ceph-mon[33787]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-06T14:24:02.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:01 smithi162 ceph-mon[33787]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-06T14:24:02.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:01 smithi142 ceph-mon[29958]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-06T14:24:02.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:01 smithi142 ceph-mon[37372]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-06T14:24:04.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:03 smithi142 ceph-mon[29958]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-06T14:24:04.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:03 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-06T14:24:04.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:03 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:24:04.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:03 smithi142 ceph-mon[37372]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-06T14:24:04.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:03 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-06T14:24:04.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:03 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:24:04.221 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:03 smithi162 ceph-mon[33787]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-06T14:24:04.221 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:03 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-06T14:24:04.221 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:03 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:24:05.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:04 smithi142 ceph-mon[29958]: Deploying daemon osd.7 on smithi162 2024-09-06T14:24:05.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:04 smithi142 ceph-mon[37372]: Deploying daemon osd.7 on smithi162 2024-09-06T14:24:05.234 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:04 smithi162 ceph-mon[33787]: Deploying daemon osd.7 on smithi162 2024-09-06T14:24:06.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:05 smithi142 ceph-mon[29958]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-06T14:24:06.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:05 smithi142 ceph-mon[37372]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-06T14:24:06.263 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:05 smithi162 ceph-mon[33787]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-06T14:24:08.103 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:07 smithi162 ceph-mon[33787]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-06T14:24:08.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:07 smithi142 ceph-mon[29958]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-06T14:24:08.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:07 smithi142 ceph-mon[37372]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-06T14:24:10.068 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:09 smithi162 ceph-mon[33787]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-06T14:24:10.068 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:09 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-06T14:24:10.068 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:09 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:10.068 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:09 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:10.458 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:09 smithi142 ceph-mon[29958]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-06T14:24:10.459 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:09 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-06T14:24:10.459 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:09 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:10.459 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:09 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:10.459 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:09 smithi142 ceph-mon[37372]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-06T14:24:10.459 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:09 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-06T14:24:10.459 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:09 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:10.459 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:09 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:12.255 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:11 smithi162 ceph-mon[33787]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-06T14:24:12.458 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:11 smithi142 ceph-mon[29958]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-06T14:24:12.458 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:11 smithi142 ceph-mon[37372]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-06T14:24:12.558 INFO:teuthology.orchestra.run.smithi162.stdout:Created osd(s) 7 on host 'smithi162' 2024-09-06T14:24:12.559 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:12.560+0000 7f2d5e7fc640 1 -- 172.21.15.162:0/620178586 <== mgr.14146 v2:172.21.15.142:6800/853729507 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f2d6802e5c0 con 0x7f2d54077530 2024-09-06T14:24:12.560 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:12.563+0000 7f2d6f70d640 1 -- 172.21.15.162:0/620178586 >> v2:172.21.15.142:6800/853729507 conn(0x7f2d54077530 msgr2=0x7f2d540799f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:12.560 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:12.563+0000 7f2d6f70d640 1 --2- 172.21.15.162:0/620178586 >> v2:172.21.15.142:6800/853729507 conn(0x7f2d54077530 0x7f2d540799f0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2d6400abe0 tx=0x7f2d6400a2d0 comp rx=0 tx=0).stop 2024-09-06T14:24:12.560 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:12.563+0000 7f2d6f70d640 1 -- 172.21.15.162:0/620178586 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2d68106280 msgr2=0x7f2d6819f9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:12.560 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:12.563+0000 7f2d6f70d640 1 --2- 172.21.15.162:0/620178586 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2d68106280 0x7f2d6819f9d0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f2d6000cce0 tx=0x7f2d6000a620 comp rx=0 tx=0).stop 2024-09-06T14:24:12.561 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:12.563+0000 7f2d6f70d640 1 -- 172.21.15.162:0/620178586 shutdown_connections 2024-09-06T14:24:12.561 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:12.563+0000 7f2d6f70d640 1 --2- 172.21.15.162:0/620178586 >> v2:172.21.15.142:6800/853729507 conn(0x7f2d54077530 0x7f2d540799f0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:12.561 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:12.563+0000 7f2d6f70d640 1 --2- 172.21.15.162:0/620178586 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2d68106280 0x7f2d6819f9d0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:12.561 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:12.563+0000 7f2d6f70d640 1 --2- 172.21.15.162:0/620178586 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2d681036d0 0x7f2d6819ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:12.561 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:12.563+0000 7f2d6f70d640 1 --2- 172.21.15.162:0/620178586 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2d6806be10 0x7f2d6819e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:12.561 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:12.563+0000 7f2d6f70d640 1 -- 172.21.15.162:0/620178586 >> 172.21.15.162:0/620178586 conn(0x7f2d680fcf20 msgr2=0x7f2d68101950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:24:12.561 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:12.563+0000 7f2d6f70d640 1 -- 172.21.15.162:0/620178586 shutdown_connections 2024-09-06T14:24:12.561 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:12.563+0000 7f2d6f70d640 1 -- 172.21.15.162:0/620178586 wait complete. 2024-09-06T14:24:13.230 DEBUG:teuthology.orchestra.run.smithi162:osd.7> sudo journalctl -f -n 0 -u ceph-ce32462e-6c5a-11ef-bce4-c7b262605968@osd.7.service 2024-09-06T14:24:13.233 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-09-06T14:24:13.233 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph osd stat -f json 2024-09-06T14:24:13.495 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:24:13.818 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:13 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:13.818 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:13 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:13.818 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:13 smithi162 ceph-mon[33787]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-06T14:24:13.818 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:13 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:13.818 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:13 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:13.847 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:13 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:13.847 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:13 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:13.847 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:13 smithi142 ceph-mon[29958]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-06T14:24:13.847 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:13 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:13.847 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:13 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:13.847 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:13 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:13.847 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:13 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:13.847 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:13 smithi142 ceph-mon[37372]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-06T14:24:13.847 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:13 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:13.848 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:13 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:14.105 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:14.072+0000 7fe0af8f0640 1 -- 172.21.15.142:0/1829897562 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe0a8104280 msgr2=0x7fe0a810ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:14.105 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:14.072+0000 7fe0af8f0640 1 --2- 172.21.15.142:0/1829897562 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe0a8104280 0x7fe0a810ac00 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fe0a0009cd0 tx=0x7fe0a002a8e0 comp rx=0 tx=0).stop 2024-09-06T14:24:14.105 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:14.072+0000 7fe0af8f0640 1 -- 172.21.15.142:0/1829897562 shutdown_connections 2024-09-06T14:24:14.105 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:14.072+0000 7fe0af8f0640 1 --2- 172.21.15.142:0/1829897562 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe0a8104280 0x7fe0a810ac00 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:14.105 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:14.072+0000 7fe0af8f0640 1 --2- 172.21.15.142:0/1829897562 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe0a81038c0 0x7fe0a8103d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:14.105 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:14.072+0000 7fe0af8f0640 1 --2- 172.21.15.142:0/1829897562 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe0a8102760 0x7fe0a8102b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:14.105 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:14.072+0000 7fe0af8f0640 1 -- 172.21.15.142:0/1829897562 >> 172.21.15.142:0/1829897562 conn(0x7fe0a80fe050 msgr2=0x7fe0a8100470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:24:14.105 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:14.072+0000 7fe0af8f0640 1 -- 172.21.15.142:0/1829897562 shutdown_connections 2024-09-06T14:24:14.105 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:14.072+0000 7fe0af8f0640 1 -- 172.21.15.142:0/1829897562 wait complete. 2024-09-06T14:24:14.105 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:14.073+0000 7fe0af8f0640 1 Processor -- start 2024-09-06T14:24:14.106 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:14.073+0000 7fe0af8f0640 1 -- start start 2024-09-06T14:24:14.106 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:14.073+0000 7fe0af8f0640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe0a8102760 0x7fe0a81981c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:14.106 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:14.073+0000 7fe0af8f0640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe0a81038c0 0x7fe0a8198810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:14.106 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:14.073+0000 7fe0af8f0640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe0a8104280 0x7fe0a8199450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:14.106 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:14.073+0000 7fe0af8f0640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fe0a810da70 con 0x7fe0a8104280 2024-09-06T14:24:14.106 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:14.073+0000 7fe0af8f0640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe0a810d8f0 con 0x7fe0a81038c0 2024-09-06T14:24:14.106 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:14.073+0000 7fe0af8f0640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7fe0a810dbf0 con 0x7fe0a8102760 2024-09-06T14:24:14.106 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:14.073+0000 7fe0ad68c640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe0a8102760 0x7fe0a81981c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:14.106 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:14.073+0000 7fe0ade8d640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe0a8104280 0x7fe0a8199450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:14.106 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:14.073+0000 7fe0ad68c640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe0a8102760 0x7fe0a81981c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:35954/0 (socket says 172.21.15.142:35954) 2024-09-06T14:24:14.106 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:14.073+0000 7fe0ad68c640 1 -- 172.21.15.142:0/1352224371 learned_addr learned my addr 172.21.15.142:0/1352224371 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:24:14.106 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:14.073+0000 7fe0ace8b640 1 --2- 172.21.15.142:0/1352224371 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe0a81038c0 0x7fe0a8198810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:14.106 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:14.074+0000 7fe0ad68c640 1 -- 172.21.15.142:0/1352224371 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe0a81038c0 msgr2=0x7fe0a8198810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:14.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:14.074+0000 7fe0ad68c640 1 --2- 172.21.15.142:0/1352224371 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe0a81038c0 0x7fe0a8198810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:14.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:14.074+0000 7fe0ad68c640 1 -- 172.21.15.142:0/1352224371 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe0a8104280 msgr2=0x7fe0a8199450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:14.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:14.074+0000 7fe0ad68c640 1 --2- 172.21.15.142:0/1352224371 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe0a8104280 0x7fe0a8199450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:14.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:14.074+0000 7fe0ad68c640 1 -- 172.21.15.142:0/1352224371 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0a8199cf0 con 0x7fe0a8102760 2024-09-06T14:24:14.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:14.074+0000 7fe0ade8d640 1 --2- 172.21.15.142:0/1352224371 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe0a8104280 0x7fe0a8199450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-06T14:24:14.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:14.074+0000 7fe0ad68c640 1 --2- 172.21.15.142:0/1352224371 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe0a8102760 0x7fe0a81981c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe0a400e9f0 tx=0x7fe0a400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:24:14.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:14.074+0000 7fe09e7fc640 1 -- 172.21.15.142:0/1352224371 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0a400cd80 con 0x7fe0a8102760 2024-09-06T14:24:14.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:14.074+0000 7fe0af8f0640 1 -- 172.21.15.142:0/1352224371 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0a8199fe0 con 0x7fe0a8102760 2024-09-06T14:24:14.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:14.074+0000 7fe09e7fc640 1 -- 172.21.15.142:0/1352224371 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7fe0a40108b0 con 0x7fe0a8102760 2024-09-06T14:24:14.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:14.074+0000 7fe09e7fc640 1 -- 172.21.15.142:0/1352224371 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0a4018870 con 0x7fe0a8102760 2024-09-06T14:24:14.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:14.074+0000 7fe0af8f0640 1 -- 172.21.15.142:0/1352224371 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0a81a06c0 con 0x7fe0a8102760 2024-09-06T14:24:14.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:14.075+0000 7fe0af8f0640 1 -- 172.21.15.142:0/1352224371 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0a8102b60 con 0x7fe0a8102760 2024-09-06T14:24:14.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:14.076+0000 7fe09e7fc640 1 -- 172.21.15.142:0/1352224371 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 15) ==== 99786+0+0 (secure 0 0 0) 0x7fe0a4010430 con 0x7fe0a8102760 2024-09-06T14:24:14.108 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:14.076+0000 7fe09e7fc640 1 --2- 172.21.15.142:0/1352224371 >> v2:172.21.15.142:6800/853729507 conn(0x7fe094077530 0x7fe0940799f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:14.108 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:14.076+0000 7fe09e7fc640 1 -- 172.21.15.142:0/1352224371 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(45..45 src has 1..45) ==== 3818+0+0 (secure 0 0 0) 0x7fe0a4099250 con 0x7fe0a8102760 2024-09-06T14:24:14.108 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:14.076+0000 7fe0ace8b640 1 --2- 172.21.15.142:0/1352224371 >> v2:172.21.15.142:6800/853729507 conn(0x7fe094077530 0x7fe0940799f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:14.108 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:14.077+0000 7fe0ace8b640 1 --2- 172.21.15.142:0/1352224371 >> v2:172.21.15.142:6800/853729507 conn(0x7fe094077530 0x7fe0940799f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fe09800bbe0 tx=0x7fe09800b390 comp rx=0 tx=0).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:24:14.108 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:14.080+0000 7fe09e7fc640 1 -- 172.21.15.142:0/1352224371 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7fe0a4062690 con 0x7fe0a8102760 2024-09-06T14:24:14.233 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:14.236+0000 7fe0af8f0640 1 -- 172.21.15.142:0/1352224371 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7fe0a802e590 con 0x7fe0a8102760 2024-09-06T14:24:14.233 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:14.236+0000 7fe09e7fc640 1 -- 172.21.15.142:0/1352224371 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v45) ==== 74+0+130 (secure 0 0 0) 0x7fe0a406b190 con 0x7fe0a8102760 2024-09-06T14:24:14.233 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:24:14.235 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:14.238+0000 7fe0af8f0640 1 -- 172.21.15.142:0/1352224371 >> v2:172.21.15.142:6800/853729507 conn(0x7fe094077530 msgr2=0x7fe0940799f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:14.235 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:14.238+0000 7fe0af8f0640 1 --2- 172.21.15.142:0/1352224371 >> v2:172.21.15.142:6800/853729507 conn(0x7fe094077530 0x7fe0940799f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fe09800bbe0 tx=0x7fe09800b390 comp rx=0 tx=0).stop 2024-09-06T14:24:14.235 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:14.238+0000 7fe0af8f0640 1 -- 172.21.15.142:0/1352224371 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe0a8102760 msgr2=0x7fe0a81981c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:14.235 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:14.238+0000 7fe0af8f0640 1 --2- 172.21.15.142:0/1352224371 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe0a8102760 0x7fe0a81981c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe0a400e9f0 tx=0x7fe0a400eeb0 comp rx=0 tx=0).stop 2024-09-06T14:24:14.235 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:14.239+0000 7fe0af8f0640 1 -- 172.21.15.142:0/1352224371 shutdown_connections 2024-09-06T14:24:14.236 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:14.239+0000 7fe0af8f0640 1 --2- 172.21.15.142:0/1352224371 >> v2:172.21.15.142:6800/853729507 conn(0x7fe094077530 0x7fe0940799f0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:14.236 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:14.239+0000 7fe0af8f0640 1 --2- 172.21.15.142:0/1352224371 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe0a8104280 0x7fe0a8199450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:14.236 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:14.239+0000 7fe0af8f0640 1 --2- 172.21.15.142:0/1352224371 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe0a81038c0 0x7fe0a8198810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:14.236 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:14.239+0000 7fe0af8f0640 1 --2- 172.21.15.142:0/1352224371 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe0a8102760 0x7fe0a81981c0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:14.236 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:14.239+0000 7fe0af8f0640 1 -- 172.21.15.142:0/1352224371 >> 172.21.15.142:0/1352224371 conn(0x7fe0a80fe050 msgr2=0x7fe0a80ffae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:24:14.236 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:14.239+0000 7fe0af8f0640 1 -- 172.21.15.142:0/1352224371 shutdown_connections 2024-09-06T14:24:14.236 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:14.239+0000 7fe0af8f0640 1 -- 172.21.15.142:0/1352224371 wait complete. 2024-09-06T14:24:14.412 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:24:14 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7[55156]: 2024-09-06T14:24:14.180+0000 7fa8abddb640 -1 osd.7 0 log_to_monitors true 2024-09-06T14:24:14.739 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:14 smithi162 ceph-mon[33787]: from='osd.7 v2:172.21.15.162:6812/1194583876' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-06T14:24:14.740 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:14 smithi162 ceph-mon[33787]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-06T14:24:14.740 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:14 smithi162 ceph-mon[33787]: from='client.? 172.21.15.142:0/1352224371' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-06T14:24:14.958 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:14 smithi142 ceph-mon[29958]: from='osd.7 v2:172.21.15.162:6812/1194583876' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-06T14:24:14.958 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:14 smithi142 ceph-mon[29958]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-06T14:24:14.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:14 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/1352224371' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-06T14:24:14.959 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:14 smithi142 ceph-mon[37372]: from='osd.7 v2:172.21.15.162:6812/1194583876' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-06T14:24:14.959 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:14 smithi142 ceph-mon[37372]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-06T14:24:14.959 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:14 smithi142 ceph-mon[37372]: from='client.? 172.21.15.142:0/1352224371' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-06T14:24:15.022 INFO:teuthology.orchestra.run.smithi142.stdout:{"epoch":45,"num_osds":8,"num_up_osds":7,"osd_up_since":1725632630,"num_in_osds":8,"osd_in_since":1725632636,"num_remapped_pgs":0} 2024-09-06T14:24:15.580 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:15 smithi162 ceph-mon[33787]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-06T14:24:15.580 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:15 smithi162 ceph-mon[33787]: from='osd.7 v2:172.21.15.162:6812/1194583876' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-06T14:24:15.580 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:15 smithi162 ceph-mon[33787]: osdmap e46: 8 total, 7 up, 8 in 2024-09-06T14:24:15.580 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:15 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-06T14:24:15.580 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:15 smithi162 ceph-mon[33787]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-06T14:24:15.580 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:15 smithi162 ceph-mon[33787]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-06T14:24:15.958 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:15 smithi142 ceph-mon[37372]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-06T14:24:15.958 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:15 smithi142 ceph-mon[37372]: from='osd.7 v2:172.21.15.162:6812/1194583876' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-06T14:24:15.959 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:15 smithi142 ceph-mon[37372]: osdmap e46: 8 total, 7 up, 8 in 2024-09-06T14:24:15.959 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:15 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-06T14:24:15.959 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:15 smithi142 ceph-mon[37372]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-06T14:24:15.959 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:15 smithi142 ceph-mon[37372]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-06T14:24:15.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:15 smithi142 ceph-mon[29958]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-06T14:24:15.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:15 smithi142 ceph-mon[29958]: from='osd.7 v2:172.21.15.162:6812/1194583876' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-06T14:24:15.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:15 smithi142 ceph-mon[29958]: osdmap e46: 8 total, 7 up, 8 in 2024-09-06T14:24:15.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:15 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-06T14:24:15.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:15 smithi142 ceph-mon[29958]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-06T14:24:15.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:15 smithi142 ceph-mon[29958]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-06T14:24:16.024 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph osd stat -f json 2024-09-06T14:24:16.280 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:24:16.413 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:24:16 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7[55156]: 2024-09-06T14:24:16.053+0000 7fa8a7d5e640 -1 osd.7 0 waiting for initial osdmap 2024-09-06T14:24:16.414 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:24:16 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7[55156]: 2024-09-06T14:24:16.061+0000 7fa89fb7d640 -1 osd.7 47 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-06T14:24:16.653 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:16 smithi142 ceph-mon[37372]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-09-06T14:24:16.653 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:16 smithi142 ceph-mon[37372]: osdmap e47: 8 total, 7 up, 8 in 2024-09-06T14:24:16.654 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:16 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-06T14:24:16.654 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:16 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-06T14:24:16.654 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:16 smithi142 ceph-mon[37372]: from='osd.7 ' entity='osd.7' 2024-09-06T14:24:16.654 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:16 smithi142 ceph-mon[29958]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-09-06T14:24:16.654 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:16 smithi142 ceph-mon[29958]: osdmap e47: 8 total, 7 up, 8 in 2024-09-06T14:24:16.654 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:16 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-06T14:24:16.654 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:16 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-06T14:24:16.654 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:16 smithi142 ceph-mon[29958]: from='osd.7 ' entity='osd.7' 2024-09-06T14:24:16.731 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:16 smithi162 ceph-mon[33787]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-09-06T14:24:16.731 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:16 smithi162 ceph-mon[33787]: osdmap e47: 8 total, 7 up, 8 in 2024-09-06T14:24:16.731 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:16 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-06T14:24:16.731 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:16 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-06T14:24:16.731 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:16 smithi162 ceph-mon[33787]: from='osd.7 ' entity='osd.7' 2024-09-06T14:24:16.734 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:16.737+0000 7f168299e640 1 -- 172.21.15.142:0/33453552 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f167c103190 msgr2=0x7f167c109b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:16.734 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:16.737+0000 7f168299e640 1 --2- 172.21.15.142:0/33453552 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f167c103190 0x7f167c109b10 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f1674009cd0 tx=0x7f167402a8e0 comp rx=0 tx=0).stop 2024-09-06T14:24:16.734 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:16.737+0000 7f168299e640 1 -- 172.21.15.142:0/33453552 shutdown_connections 2024-09-06T14:24:16.734 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:16.737+0000 7f168299e640 1 --2- 172.21.15.142:0/33453552 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f167c103190 0x7f167c109b10 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:16.734 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:16.737+0000 7f168299e640 1 --2- 172.21.15.142:0/33453552 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f167c1027d0 0x7f167c102c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:16.734 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:16.737+0000 7f168299e640 1 --2- 172.21.15.142:0/33453552 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f167c101670 0x7f167c101a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:16.734 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:16.737+0000 7f168299e640 1 -- 172.21.15.142:0/33453552 >> 172.21.15.142:0/33453552 conn(0x7f167c0fcf20 msgr2=0x7f167c0ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:24:16.734 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:16.737+0000 7f168299e640 1 -- 172.21.15.142:0/33453552 shutdown_connections 2024-09-06T14:24:16.734 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:16.737+0000 7f168299e640 1 -- 172.21.15.142:0/33453552 wait complete. 2024-09-06T14:24:16.736 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:16.738+0000 7f168299e640 1 Processor -- start 2024-09-06T14:24:16.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:16.738+0000 7f168299e640 1 -- start start 2024-09-06T14:24:16.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:16.738+0000 7f168299e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f167c101670 0x7f167c19a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:16.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:16.738+0000 7f168299e640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f167c1027d0 0x7f167c19a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:16.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:16.738+0000 7f168299e640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f167c103190 0x7f167c19b630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:16.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:16.738+0000 7f168299e640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f167c10b930 con 0x7f167c103190 2024-09-06T14:24:16.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:16.738+0000 7f168299e640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f167c10b7b0 con 0x7f167c101670 2024-09-06T14:24:16.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:16.738+0000 7f168299e640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7f167c10bab0 con 0x7f167c1027d0 2024-09-06T14:24:16.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:16.738+0000 7f167b7fe640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f167c1027d0 0x7f167c19a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:16.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:16.738+0000 7f1680f3b640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f167c103190 0x7f167c19b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:16.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:16.738+0000 7f167b7fe640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f167c1027d0 0x7f167c19a9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:35978/0 (socket says 172.21.15.142:35978) 2024-09-06T14:24:16.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:16.738+0000 7f167b7fe640 1 -- 172.21.15.142:0/3338671929 learned_addr learned my addr 172.21.15.142:0/3338671929 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:24:16.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:16.738+0000 7f167bfff640 1 --2- 172.21.15.142:0/3338671929 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f167c101670 0x7f167c19a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:16.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:16.739+0000 7f167b7fe640 1 -- 172.21.15.142:0/3338671929 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f167c101670 msgr2=0x7f167c19a3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:16.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:16.739+0000 7f167b7fe640 1 --2- 172.21.15.142:0/3338671929 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f167c101670 0x7f167c19a3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:16.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:16.739+0000 7f167b7fe640 1 -- 172.21.15.142:0/3338671929 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f167c103190 msgr2=0x7f167c19b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:16.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:16.739+0000 7f167b7fe640 1 --2- 172.21.15.142:0/3338671929 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f167c103190 0x7f167c19b630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:16.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:16.739+0000 7f167b7fe640 1 -- 172.21.15.142:0/3338671929 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f167c19bed0 con 0x7f167c1027d0 2024-09-06T14:24:16.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:16.739+0000 7f1680f3b640 1 --2- 172.21.15.142:0/3338671929 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f167c103190 0x7f167c19b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-06T14:24:16.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:16.739+0000 7f167b7fe640 1 --2- 172.21.15.142:0/3338671929 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f167c1027d0 0x7f167c19a9f0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f167000ea60 tx=0x7f167000ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:24:16.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:16.739+0000 7f16797fa640 1 -- 172.21.15.142:0/3338671929 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f167000cde0 con 0x7f167c1027d0 2024-09-06T14:24:16.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:16.739+0000 7f168299e640 1 -- 172.21.15.142:0/3338671929 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f167c19c1c0 con 0x7f167c1027d0 2024-09-06T14:24:16.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:16.739+0000 7f16797fa640 1 -- 172.21.15.142:0/3338671929 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7f16700035f0 con 0x7f167c1027d0 2024-09-06T14:24:16.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:16.739+0000 7f16797fa640 1 -- 172.21.15.142:0/3338671929 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16700044a0 con 0x7f167c1027d0 2024-09-06T14:24:16.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:16.739+0000 7f168299e640 1 -- 172.21.15.142:0/3338671929 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f167c1a28a0 con 0x7f167c1027d0 2024-09-06T14:24:16.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:16.740+0000 7f168299e640 1 -- 172.21.15.142:0/3338671929 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f167c101a70 con 0x7f167c1027d0 2024-09-06T14:24:16.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:16.741+0000 7f16797fa640 1 -- 172.21.15.142:0/3338671929 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 15) ==== 99786+0+0 (secure 0 0 0) 0x7f1670004000 con 0x7f167c1027d0 2024-09-06T14:24:16.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:16.741+0000 7f16797fa640 1 --2- 172.21.15.142:0/3338671929 >> v2:172.21.15.142:6800/853729507 conn(0x7f1664077530 0x7f16640799f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:16.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:16.741+0000 7f167bfff640 1 --2- 172.21.15.142:0/3338671929 >> v2:172.21.15.142:6800/853729507 conn(0x7f1664077530 0x7f16640799f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:16.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:16.741+0000 7f16797fa640 1 -- 172.21.15.142:0/3338671929 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(48..48 src has 1..48) ==== 4028+0+0 (secure 0 0 0) 0x7f16700993b0 con 0x7f167c1027d0 2024-09-06T14:24:16.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:16.742+0000 7f167bfff640 1 --2- 172.21.15.142:0/3338671929 >> v2:172.21.15.142:6800/853729507 conn(0x7f1664077530 0x7f16640799f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f166c00b3c0 tx=0x7f166c00d040 comp rx=0 tx=0).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:24:16.741 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:16.745+0000 7f16797fa640 1 -- 172.21.15.142:0/3338671929 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7f1670062660 con 0x7f167c1027d0 2024-09-06T14:24:16.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:16.933+0000 7f168299e640 1 -- 172.21.15.142:0/3338671929 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f167c02e5c0 con 0x7f167c1027d0 2024-09-06T14:24:16.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:16.934+0000 7f16797fa640 1 -- 172.21.15.142:0/3338671929 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v48) ==== 74+0+130 (secure 0 0 0) 0x7f1670066e20 con 0x7f167c1027d0 2024-09-06T14:24:16.931 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:24:16.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:16.936+0000 7f168299e640 1 -- 172.21.15.142:0/3338671929 >> v2:172.21.15.142:6800/853729507 conn(0x7f1664077530 msgr2=0x7f16640799f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:16.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:16.936+0000 7f168299e640 1 --2- 172.21.15.142:0/3338671929 >> v2:172.21.15.142:6800/853729507 conn(0x7f1664077530 0x7f16640799f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f166c00b3c0 tx=0x7f166c00d040 comp rx=0 tx=0).stop 2024-09-06T14:24:16.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:16.936+0000 7f168299e640 1 -- 172.21.15.142:0/3338671929 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f167c1027d0 msgr2=0x7f167c19a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:16.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:16.936+0000 7f168299e640 1 --2- 172.21.15.142:0/3338671929 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f167c1027d0 0x7f167c19a9f0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f167000ea60 tx=0x7f167000ef20 comp rx=0 tx=0).stop 2024-09-06T14:24:16.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:16.936+0000 7f168299e640 1 -- 172.21.15.142:0/3338671929 shutdown_connections 2024-09-06T14:24:16.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:16.936+0000 7f168299e640 1 --2- 172.21.15.142:0/3338671929 >> v2:172.21.15.142:6800/853729507 conn(0x7f1664077530 0x7f16640799f0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:16.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:16.936+0000 7f168299e640 1 --2- 172.21.15.142:0/3338671929 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f167c103190 0x7f167c19b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:16.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:16.936+0000 7f168299e640 1 --2- 172.21.15.142:0/3338671929 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f167c1027d0 0x7f167c19a9f0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:16.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:16.936+0000 7f168299e640 1 --2- 172.21.15.142:0/3338671929 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f167c101670 0x7f167c19a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:16.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:16.936+0000 7f168299e640 1 -- 172.21.15.142:0/3338671929 >> 172.21.15.142:0/3338671929 conn(0x7f167c0fcf20 msgr2=0x7f167c190fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:24:16.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:16.936+0000 7f168299e640 1 -- 172.21.15.142:0/3338671929 shutdown_connections 2024-09-06T14:24:16.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:16.936+0000 7f168299e640 1 -- 172.21.15.142:0/3338671929 wait complete. 2024-09-06T14:24:17.683 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:17 smithi142 ceph-mon[29958]: purged_snaps scrub starts 2024-09-06T14:24:17.684 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:17 smithi142 ceph-mon[29958]: purged_snaps scrub ok 2024-09-06T14:24:17.684 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:17 smithi142 ceph-mon[29958]: osd.7 v2:172.21.15.162:6812/1194583876 boot 2024-09-06T14:24:17.684 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:17 smithi142 ceph-mon[29958]: osdmap e48: 8 total, 8 up, 8 in 2024-09-06T14:24:17.684 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:17 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-06T14:24:17.684 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:17 smithi142 ceph-mon[29958]: Detected new or changed devices on smithi162 2024-09-06T14:24:17.684 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:17 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:17.684 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:17 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:17.684 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:17 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-06T14:24:17.684 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:17 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-06T14:24:17.684 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:17 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-06T14:24:17.684 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:17 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-06T14:24:17.684 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:17 smithi142 ceph-mon[29958]: Adjusting osd_memory_target on smithi162 to 4272M 2024-09-06T14:24:17.684 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:17 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:17.685 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:17 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:24:17.685 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:17 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-06T14:24:17.685 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:17 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:17.685 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:17 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-06T14:24:17.685 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:17 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:24:17.685 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:17 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-06T14:24:17.685 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:17 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:17.685 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:17 smithi142 ceph-mon[29958]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-09-06T14:24:17.685 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:17 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/3338671929' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-06T14:24:17.685 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:17 smithi142 ceph-mon[29958]: osdmap e49: 8 total, 8 up, 8 in 2024-09-06T14:24:17.685 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:17 smithi142 ceph-mon[37372]: purged_snaps scrub starts 2024-09-06T14:24:17.686 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:17 smithi142 ceph-mon[37372]: purged_snaps scrub ok 2024-09-06T14:24:17.686 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:17 smithi142 ceph-mon[37372]: osd.7 v2:172.21.15.162:6812/1194583876 boot 2024-09-06T14:24:17.686 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:17 smithi142 ceph-mon[37372]: osdmap e48: 8 total, 8 up, 8 in 2024-09-06T14:24:17.686 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:17 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-06T14:24:17.686 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:17 smithi142 ceph-mon[37372]: Detected new or changed devices on smithi162 2024-09-06T14:24:17.686 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:17 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:17.686 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:17 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:17.686 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:17 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-06T14:24:17.686 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:17 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-06T14:24:17.686 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:17 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-06T14:24:17.686 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:17 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-06T14:24:17.686 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:17 smithi142 ceph-mon[37372]: Adjusting osd_memory_target on smithi162 to 4272M 2024-09-06T14:24:17.686 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:17 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:17.687 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:17 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:24:17.687 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:17 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-06T14:24:17.687 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:17 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:17.687 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:17 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-06T14:24:17.687 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:17 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:24:17.687 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:17 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-06T14:24:17.687 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:17 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:17.687 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:17 smithi142 ceph-mon[37372]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-09-06T14:24:17.687 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:17 smithi142 ceph-mon[37372]: from='client.? 172.21.15.142:0/3338671929' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-06T14:24:17.687 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:17 smithi142 ceph-mon[37372]: osdmap e49: 8 total, 8 up, 8 in 2024-09-06T14:24:17.713 INFO:teuthology.orchestra.run.smithi142.stdout:{"epoch":48,"num_osds":8,"num_up_osds":8,"osd_up_since":1725632656,"num_in_osds":8,"osd_in_since":1725632636,"num_remapped_pgs":1} 2024-09-06T14:24:17.714 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph osd dump --format=json 2024-09-06T14:24:17.960 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:24:18.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:17 smithi162 ceph-mon[33787]: purged_snaps scrub starts 2024-09-06T14:24:18.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:17 smithi162 ceph-mon[33787]: purged_snaps scrub ok 2024-09-06T14:24:18.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:17 smithi162 ceph-mon[33787]: osd.7 v2:172.21.15.162:6812/1194583876 boot 2024-09-06T14:24:18.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:17 smithi162 ceph-mon[33787]: osdmap e48: 8 total, 8 up, 8 in 2024-09-06T14:24:18.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:17 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-06T14:24:18.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:17 smithi162 ceph-mon[33787]: Detected new or changed devices on smithi162 2024-09-06T14:24:18.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:17 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:18.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:17 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:18.048 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:17 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-06T14:24:18.048 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:17 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-06T14:24:18.048 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:17 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-06T14:24:18.048 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:17 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-06T14:24:18.048 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:17 smithi162 ceph-mon[33787]: Adjusting osd_memory_target on smithi162 to 4272M 2024-09-06T14:24:18.048 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:17 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:18.048 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:17 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:24:18.048 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:17 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-06T14:24:18.048 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:17 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:18.048 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:17 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-06T14:24:18.048 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:17 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:24:18.048 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:17 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-06T14:24:18.049 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:17 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:18.049 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:17 smithi162 ceph-mon[33787]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-09-06T14:24:18.049 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:17 smithi162 ceph-mon[33787]: from='client.? 172.21.15.142:0/3338671929' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-06T14:24:18.049 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:17 smithi162 ceph-mon[33787]: osdmap e49: 8 total, 8 up, 8 in 2024-09-06T14:24:18.420 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:18.411+0000 7fc53a3f6640 1 -- 172.21.15.142:0/1495822 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc5341073d0 msgr2=0x7fc534107810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:18.420 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:18.411+0000 7fc53a3f6640 1 --2- 172.21.15.142:0/1495822 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc5341073d0 0x7fc534107810 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc51c009900 tx=0x7fc51c02a410 comp rx=0 tx=0).stop 2024-09-06T14:24:18.420 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:18.411+0000 7fc53a3f6640 1 -- 172.21.15.142:0/1495822 shutdown_connections 2024-09-06T14:24:18.420 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:18.411+0000 7fc53a3f6640 1 --2- 172.21.15.142:0/1495822 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc534107d50 0x7fc53410e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:18.420 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:18.411+0000 7fc53a3f6640 1 --2- 172.21.15.142:0/1495822 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc5341073d0 0x7fc534107810 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:18.420 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:18.411+0000 7fc53a3f6640 1 --2- 172.21.15.142:0/1495822 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc5341019c0 0x7fc534101da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:18.420 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:18.411+0000 7fc53a3f6640 1 -- 172.21.15.142:0/1495822 >> 172.21.15.142:0/1495822 conn(0x7fc5340fd430 msgr2=0x7fc5340ff850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:24:18.420 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:18.411+0000 7fc53a3f6640 1 -- 172.21.15.142:0/1495822 shutdown_connections 2024-09-06T14:24:18.420 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:18.411+0000 7fc53a3f6640 1 -- 172.21.15.142:0/1495822 wait complete. 2024-09-06T14:24:18.420 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:18.411+0000 7fc53a3f6640 1 Processor -- start 2024-09-06T14:24:18.421 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:18.411+0000 7fc53a3f6640 1 -- start start 2024-09-06T14:24:18.421 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:18.411+0000 7fc53a3f6640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc5341019c0 0x7fc534197560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:18.421 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:18.411+0000 7fc53a3f6640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc5341073d0 0x7fc534197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:18.421 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:18.411+0000 7fc53a3f6640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc534107d50 0x7fc53419ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:18.421 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:18.411+0000 7fc53a3f6640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fc5341110b0 con 0x7fc5341073d0 2024-09-06T14:24:18.421 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:18.411+0000 7fc53a3f6640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc534110f30 con 0x7fc5341019c0 2024-09-06T14:24:18.421 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:18.411+0000 7fc53a3f6640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7fc534111230 con 0x7fc534107d50 2024-09-06T14:24:18.421 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:18.412+0000 7fc538993640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc534107d50 0x7fc53419ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:18.421 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:18.412+0000 7fc52bfff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc5341073d0 0x7fc534197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:18.421 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:18.412+0000 7fc538993640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc534107d50 0x7fc53419ce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:54660/0 (socket says 172.21.15.142:54660) 2024-09-06T14:24:18.421 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:18.412+0000 7fc538993640 1 -- 172.21.15.142:0/1139507415 learned_addr learned my addr 172.21.15.142:0/1139507415 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:24:18.421 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:18.412+0000 7fc52bfff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc5341073d0 0x7fc534197aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:59106/0 (socket says 172.21.15.142:59106) 2024-09-06T14:24:18.421 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:18.412+0000 7fc533fff640 1 --2- 172.21.15.142:0/1139507415 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc5341019c0 0x7fc534197560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:18.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:18.412+0000 7fc538993640 1 -- 172.21.15.142:0/1139507415 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc5341019c0 msgr2=0x7fc534197560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:18.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:18.412+0000 7fc538993640 1 --2- 172.21.15.142:0/1139507415 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc5341019c0 0x7fc534197560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:18.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:18.412+0000 7fc538993640 1 -- 172.21.15.142:0/1139507415 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc5341073d0 msgr2=0x7fc534197aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:18.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:18.412+0000 7fc538993640 1 --2- 172.21.15.142:0/1139507415 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc5341073d0 0x7fc534197aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:18.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:18.412+0000 7fc538993640 1 -- 172.21.15.142:0/1139507415 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc53419d610 con 0x7fc534107d50 2024-09-06T14:24:18.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:18.412+0000 7fc52bfff640 1 --2- 172.21.15.142:0/1139507415 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc5341073d0 0x7fc534197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-06T14:24:18.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:18.412+0000 7fc538993640 1 --2- 172.21.15.142:0/1139507415 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc534107d50 0x7fc53419ce40 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fc52c007cb0 tx=0x7fc52c00e420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:24:18.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:18.412+0000 7fc531ffb640 1 -- 172.21.15.142:0/1139507415 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc52c00eb30 con 0x7fc534107d50 2024-09-06T14:24:18.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:18.412+0000 7fc531ffb640 1 -- 172.21.15.142:0/1139507415 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7fc52c00ecd0 con 0x7fc534107d50 2024-09-06T14:24:18.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:18.412+0000 7fc53a3f6640 1 -- 172.21.15.142:0/1139507415 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc53419d900 con 0x7fc534107d50 2024-09-06T14:24:18.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:18.412+0000 7fc531ffb640 1 -- 172.21.15.142:0/1139507415 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc52c0153f0 con 0x7fc534107d50 2024-09-06T14:24:18.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:18.412+0000 7fc53a3f6640 1 -- 172.21.15.142:0/1139507415 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc53406a7e0 con 0x7fc534107d50 2024-09-06T14:24:18.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:18.413+0000 7fc53a3f6640 1 -- 172.21.15.142:0/1139507415 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc534101da0 con 0x7fc534107d50 2024-09-06T14:24:18.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:18.414+0000 7fc531ffb640 1 -- 172.21.15.142:0/1139507415 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 15) ==== 99786+0+0 (secure 0 0 0) 0x7fc52c00a440 con 0x7fc534107d50 2024-09-06T14:24:18.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:18.414+0000 7fc531ffb640 1 --2- 172.21.15.142:0/1139507415 >> v2:172.21.15.142:6800/853729507 conn(0x7fc520077530 0x7fc5200799f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:18.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:18.415+0000 7fc531ffb640 1 -- 172.21.15.142:0/1139507415 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(49..49 src has 1..49) ==== 3995+0+0 (secure 0 0 0) 0x7fc52c09ac00 con 0x7fc534107d50 2024-09-06T14:24:18.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:18.415+0000 7fc533fff640 1 --2- 172.21.15.142:0/1139507415 >> v2:172.21.15.142:6800/853729507 conn(0x7fc520077530 0x7fc5200799f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:18.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:18.415+0000 7fc533fff640 1 --2- 172.21.15.142:0/1139507415 >> v2:172.21.15.142:6800/853729507 conn(0x7fc520077530 0x7fc5200799f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fc5240095f0 tx=0x7fc524005f90 comp rx=0 tx=0).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:24:18.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:18.418+0000 7fc531ffb640 1 -- 172.21.15.142:0/1139507415 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7fc52c064520 con 0x7fc534107d50 2024-09-06T14:24:18.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:18.572+0000 7fc53a3f6640 1 -- 172.21.15.142:0/1139507415 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc53406b200 con 0x7fc534107d50 2024-09-06T14:24:18.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:18.572+0000 7fc531ffb640 1 -- 172.21.15.142:0/1139507415 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v49) ==== 74+0+11441 (secure 0 0 0) 0x7fc52c063e20 con 0x7fc534107d50 2024-09-06T14:24:18.569 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:24:18.569 INFO:teuthology.orchestra.run.smithi142.stdout:{"epoch":49,"fsid":"ce32462e-6c5a-11ef-bce4-c7b262605968","created":"2024-09-06T14:19:13.103278+0000","modified":"2024-09-06T14:24:17.579050+0000","last_up_change":"2024-09-06T14:24:16.577635+0000","last_in_change":"2024-09-06T14:23:56.281210+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-06T14:22:11.149139+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7823801-9043-4aaf-a680-046c5b16f18b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6801","nonce":1649056378}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":1649056378}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":1649056378}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6803","nonce":1649056378}]},"public_addr":"172.21.15.142:6801/1649056378","cluster_addr":"172.21.15.142:6802/1649056378","heartbeat_back_addr":"172.21.15.142:6804/1649056378","heartbeat_front_addr":"172.21.15.142:6803/1649056378","state":["exists","up"]},{"osd":1,"uuid":"364b1ffb-bdda-4b70-bd6c-3bcb73110359","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6805","nonce":599802221}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":599802221}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":599802221}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6807","nonce":599802221}]},"public_addr":"172.21.15.142:6805/599802221","cluster_addr":"172.21.15.142:6806/599802221","heartbeat_back_addr":"172.21.15.142:6808/599802221","heartbeat_front_addr":"172.21.15.142:6807/599802221","state":["exists","up"]},{"osd":2,"uuid":"d7b4c863-74aa-476b-9377-8262e2c77ae3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6809","nonce":1302270091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":1302270091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":1302270091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6811","nonce":1302270091}]},"public_addr":"172.21.15.142:6809/1302270091","cluster_addr":"172.21.15.142:6810/1302270091","heartbeat_back_addr":"172.21.15.142:6812/1302270091","heartbeat_front_addr":"172.21.15.142:6811/1302270091","state":["exists","up"]},{"osd":3,"uuid":"d8080de9-24b0-4d10-b745-ec2a23f80e49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6813","nonce":2917907718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":2917907718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":2917907718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6815","nonce":2917907718}]},"public_addr":"172.21.15.142:6813/2917907718","cluster_addr":"172.21.15.142:6814/2917907718","heartbeat_back_addr":"172.21.15.142:6816/2917907718","heartbeat_front_addr":"172.21.15.142:6815/2917907718","state":["exists","up"]},{"osd":4,"uuid":"d8c6b71c-01ef-44a5-b78b-49ecce11f357","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":2665827777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6801","nonce":2665827777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6803","nonce":2665827777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":2665827777}]},"public_addr":"172.21.15.162:6800/2665827777","cluster_addr":"172.21.15.162:6801/2665827777","heartbeat_back_addr":"172.21.15.162:6803/2665827777","heartbeat_front_addr":"172.21.15.162:6802/2665827777","state":["exists","up"]},{"osd":5,"uuid":"9215d7b7-c268-44bb-b688-74ab1576d1c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":3404139106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6805","nonce":3404139106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6807","nonce":3404139106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":3404139106}]},"public_addr":"172.21.15.162:6804/3404139106","cluster_addr":"172.21.15.162:6805/3404139106","heartbeat_back_addr":"172.21.15.162:6807/3404139106","heartbeat_front_addr":"172.21.15.162:6806/3404139106","state":["exists","up"]},{"osd":6,"uuid":"457940ad-3624-4613-98ce-817baa5f2d1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1155174053}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6809","nonce":1155174053}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6811","nonce":1155174053}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1155174053}]},"public_addr":"172.21.15.162:6808/1155174053","cluster_addr":"172.21.15.162:6809/1155174053","heartbeat_back_addr":"172.21.15.162:6811/1155174053","heartbeat_front_addr":"172.21.15.162:6810/1155174053","state":["exists","up"]},{"osd":7,"uuid":"b92b364c-849e-4971-be37-6c3f440e7cec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1194583876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6813","nonce":1194583876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6815","nonce":1194583876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1194583876}]},"public_addr":"172.21.15.162:6812/1194583876","cluster_addr":"172.21.15.162:6813/1194583876","heartbeat_back_addr":"172.21.15.162:6815/1194583876","heartbeat_front_addr":"172.21.15.162:6814/1194583876","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-06T14:21:14.998819+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-06T14:21:42.692114+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-06T14:22:09.477210+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-06T14:22:35.347799+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-06T14:22:58.792921+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-06T14:23:23.876765+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-06T14:23:49.484604+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-06T14:24:15.163203+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.142:0/516816086":"2024-09-07T14:19:58.842070+0000","172.21.15.142:6800/3892080990":"2024-09-07T14:19:58.842070+0000","172.21.15.142:0/3098992975":"2024-09-07T14:19:58.842070+0000","172.21.15.142:0/3489797750":"2024-09-07T14:19:36.083574+0000","172.21.15.142:6800/1202860895":"2024-09-07T14:19:36.083574+0000","172.21.15.142:0/382683652":"2024-09-07T14:19:36.083574+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-06T14:24:18.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:18.574+0000 7fc53a3f6640 1 -- 172.21.15.142:0/1139507415 >> v2:172.21.15.142:6800/853729507 conn(0x7fc520077530 msgr2=0x7fc5200799f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:18.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:18.574+0000 7fc53a3f6640 1 --2- 172.21.15.142:0/1139507415 >> v2:172.21.15.142:6800/853729507 conn(0x7fc520077530 0x7fc5200799f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fc5240095f0 tx=0x7fc524005f90 comp rx=0 tx=0).stop 2024-09-06T14:24:18.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:18.574+0000 7fc53a3f6640 1 -- 172.21.15.142:0/1139507415 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc534107d50 msgr2=0x7fc53419ce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:18.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:18.574+0000 7fc53a3f6640 1 --2- 172.21.15.142:0/1139507415 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc534107d50 0x7fc53419ce40 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fc52c007cb0 tx=0x7fc52c00e420 comp rx=0 tx=0).stop 2024-09-06T14:24:18.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:18.574+0000 7fc53a3f6640 1 -- 172.21.15.142:0/1139507415 shutdown_connections 2024-09-06T14:24:18.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:18.574+0000 7fc53a3f6640 1 --2- 172.21.15.142:0/1139507415 >> v2:172.21.15.142:6800/853729507 conn(0x7fc520077530 0x7fc5200799f0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:18.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:18.574+0000 7fc53a3f6640 1 --2- 172.21.15.142:0/1139507415 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc534107d50 0x7fc53419ce40 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:18.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:18.574+0000 7fc53a3f6640 1 --2- 172.21.15.142:0/1139507415 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc5341073d0 0x7fc534197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:18.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:18.574+0000 7fc53a3f6640 1 --2- 172.21.15.142:0/1139507415 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc5341019c0 0x7fc534197560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:18.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:18.574+0000 7fc53a3f6640 1 -- 172.21.15.142:0/1139507415 >> 172.21.15.142:0/1139507415 conn(0x7fc5340fd430 msgr2=0x7fc5340ff820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:24:18.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:18.574+0000 7fc53a3f6640 1 -- 172.21.15.142:0/1139507415 shutdown_connections 2024-09-06T14:24:18.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:18.574+0000 7fc53a3f6640 1 -- 172.21.15.142:0/1139507415 wait complete. 2024-09-06T14:24:19.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:18 smithi142 ceph-mon[37372]: from='client.? 172.21.15.142:0/1139507415' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-06T14:24:19.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:18 smithi142 ceph-mon[37372]: osdmap e50: 8 total, 8 up, 8 in 2024-09-06T14:24:19.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:18 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/1139507415' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-06T14:24:19.210 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:18 smithi142 ceph-mon[29958]: osdmap e50: 8 total, 8 up, 8 in 2024-09-06T14:24:19.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:18 smithi162 ceph-mon[33787]: from='client.? 172.21.15.142:0/1139507415' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-06T14:24:19.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:18 smithi162 ceph-mon[33787]: osdmap e50: 8 total, 8 up, 8 in 2024-09-06T14:24:19.353 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-06T14:22:11.149139+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-06T14:24:19.354 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-09-06T14:24:19.597 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:24:19.969 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:19 smithi142 ceph-mon[29958]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-09-06T14:24:19.969 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:19 smithi142 ceph-mon[37372]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-09-06T14:24:20.060 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:20.053+0000 7fbd69bd1640 1 -- 172.21.15.142:0/2803102468 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbd641019e0 msgr2=0x7fbd64101dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:20.060 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:20.053+0000 7fbd69bd1640 1 --2- 172.21.15.142:0/2803102468 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbd641019e0 0x7fbd64101dc0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fbd54009900 tx=0x7fbd5402a410 comp rx=0 tx=0).stop 2024-09-06T14:24:20.061 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:20.053+0000 7fbd69bd1640 1 -- 172.21.15.142:0/2803102468 shutdown_connections 2024-09-06T14:24:20.061 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:20.053+0000 7fbd69bd1640 1 --2- 172.21.15.142:0/2803102468 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbd64107d70 0x7fbd6410e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:20.061 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:20.053+0000 7fbd69bd1640 1 --2- 172.21.15.142:0/2803102468 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbd641073f0 0x7fbd64107830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:20.061 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:20.053+0000 7fbd69bd1640 1 --2- 172.21.15.142:0/2803102468 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbd641019e0 0x7fbd64101dc0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:20.061 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:20.053+0000 7fbd69bd1640 1 -- 172.21.15.142:0/2803102468 >> 172.21.15.142:0/2803102468 conn(0x7fbd640fd450 msgr2=0x7fbd640ff870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:24:20.061 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:20.053+0000 7fbd69bd1640 1 -- 172.21.15.142:0/2803102468 shutdown_connections 2024-09-06T14:24:20.061 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:20.053+0000 7fbd69bd1640 1 -- 172.21.15.142:0/2803102468 wait complete. 2024-09-06T14:24:20.061 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:20.053+0000 7fbd69bd1640 1 Processor -- start 2024-09-06T14:24:20.061 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:20.053+0000 7fbd69bd1640 1 -- start start 2024-09-06T14:24:20.061 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:20.054+0000 7fbd69bd1640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbd641019e0 0x7fbd64197650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:20.061 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:20.054+0000 7fbd69bd1640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbd641073f0 0x7fbd64197b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:20.061 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:20.054+0000 7fbd69bd1640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbd64107d70 0x7fbd6419cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:20.061 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:20.054+0000 7fbd69bd1640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fbd64111140 con 0x7fbd641019e0 2024-09-06T14:24:20.062 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:20.054+0000 7fbd69bd1640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fbd64110fc0 con 0x7fbd641073f0 2024-09-06T14:24:20.062 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:20.054+0000 7fbd69bd1640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7fbd641112c0 con 0x7fbd64107d70 2024-09-06T14:24:20.062 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:20.054+0000 7fbd637fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbd641019e0 0x7fbd64197650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:20.062 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:20.054+0000 7fbd637fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbd641019e0 0x7fbd64197650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:59122/0 (socket says 172.21.15.142:59122) 2024-09-06T14:24:20.062 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:20.054+0000 7fbd637fe640 1 -- 172.21.15.142:0/455302719 learned_addr learned my addr 172.21.15.142:0/455302719 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:24:20.062 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:20.054+0000 7fbd63fff640 1 --2- 172.21.15.142:0/455302719 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbd64107d70 0x7fbd6419cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:20.062 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:20.054+0000 7fbd5affd640 1 --2- 172.21.15.142:0/455302719 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbd641073f0 0x7fbd64197b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:20.062 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:20.054+0000 7fbd637fe640 1 -- 172.21.15.142:0/455302719 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbd64107d70 msgr2=0x7fbd6419cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:20.062 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:20.054+0000 7fbd637fe640 1 --2- 172.21.15.142:0/455302719 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbd64107d70 0x7fbd6419cf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:20.062 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:20.054+0000 7fbd637fe640 1 -- 172.21.15.142:0/455302719 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbd641073f0 msgr2=0x7fbd64197b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:20.062 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:20.054+0000 7fbd637fe640 1 --2- 172.21.15.142:0/455302719 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbd641073f0 0x7fbd64197b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:20.062 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:20.054+0000 7fbd637fe640 1 -- 172.21.15.142:0/455302719 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd6419d700 con 0x7fbd641019e0 2024-09-06T14:24:20.062 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:20.054+0000 7fbd637fe640 1 --2- 172.21.15.142:0/455302719 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbd641019e0 0x7fbd64197650 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fbd54005940 tx=0x7fbd54032c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:24:20.062 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:20.054+0000 7fbd617fa640 1 -- 172.21.15.142:0/455302719 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd5403f030 con 0x7fbd641019e0 2024-09-06T14:24:20.063 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:20.054+0000 7fbd617fa640 1 -- 172.21.15.142:0/455302719 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7fbd54005480 con 0x7fbd641019e0 2024-09-06T14:24:20.063 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:20.054+0000 7fbd69bd1640 1 -- 172.21.15.142:0/455302719 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd6419d990 con 0x7fbd641019e0 2024-09-06T14:24:20.063 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:20.054+0000 7fbd617fa640 1 -- 172.21.15.142:0/455302719 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd54035070 con 0x7fbd641019e0 2024-09-06T14:24:20.063 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:20.055+0000 7fbd69bd1640 1 -- 172.21.15.142:0/455302719 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd641a40b0 con 0x7fbd641019e0 2024-09-06T14:24:20.063 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:20.055+0000 7fbd69bd1640 1 -- 172.21.15.142:0/455302719 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd64101dc0 con 0x7fbd641019e0 2024-09-06T14:24:20.063 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:20.056+0000 7fbd617fa640 1 -- 172.21.15.142:0/455302719 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 15) ==== 99786+0+0 (secure 0 0 0) 0x7fbd54042030 con 0x7fbd641019e0 2024-09-06T14:24:20.063 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:20.057+0000 7fbd617fa640 1 --2- 172.21.15.142:0/455302719 >> v2:172.21.15.142:6800/853729507 conn(0x7fbd50077530 0x7fbd500799f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:20.063 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:20.057+0000 7fbd5affd640 1 --2- 172.21.15.142:0/455302719 >> v2:172.21.15.142:6800/853729507 conn(0x7fbd50077530 0x7fbd500799f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:20.063 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:20.057+0000 7fbd617fa640 1 -- 172.21.15.142:0/455302719 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 3995+0+0 (secure 0 0 0) 0x7fbd540b9460 con 0x7fbd641019e0 2024-09-06T14:24:20.063 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:20.057+0000 7fbd5affd640 1 --2- 172.21.15.142:0/455302719 >> v2:172.21.15.142:6800/853729507 conn(0x7fbd50077530 0x7fbd500799f0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fbd4c005fd0 tx=0x7fbd4c005850 comp rx=0 tx=0).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:24:20.063 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:20.061+0000 7fbd617fa640 1 -- 172.21.15.142:0/455302719 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7fbd54082730 con 0x7fbd641019e0 2024-09-06T14:24:20.207 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:20.211+0000 7fbd69bd1640 1 -- 172.21.15.142:0/455302719 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fbd64109b50 con 0x7fbd641019e0 2024-09-06T14:24:20.208 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:20.211+0000 7fbd617fa640 1 -- 172.21.15.142:0/455302719 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v50) ==== 93+0+10 (secure 0 0 0) 0x7fbd5408b230 con 0x7fbd641019e0 2024-09-06T14:24:20.208 INFO:teuthology.orchestra.run.smithi142.stdout:pg_num: 1 2024-09-06T14:24:20.210 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:20.213+0000 7fbd69bd1640 1 -- 172.21.15.142:0/455302719 >> v2:172.21.15.142:6800/853729507 conn(0x7fbd50077530 msgr2=0x7fbd500799f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:20.210 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:20.213+0000 7fbd69bd1640 1 --2- 172.21.15.142:0/455302719 >> v2:172.21.15.142:6800/853729507 conn(0x7fbd50077530 0x7fbd500799f0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fbd4c005fd0 tx=0x7fbd4c005850 comp rx=0 tx=0).stop 2024-09-06T14:24:20.210 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:20.213+0000 7fbd69bd1640 1 -- 172.21.15.142:0/455302719 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbd641019e0 msgr2=0x7fbd64197650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:20.210 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:20.213+0000 7fbd69bd1640 1 --2- 172.21.15.142:0/455302719 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbd641019e0 0x7fbd64197650 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fbd54005940 tx=0x7fbd54032c70 comp rx=0 tx=0).stop 2024-09-06T14:24:20.210 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:20.213+0000 7fbd69bd1640 1 -- 172.21.15.142:0/455302719 shutdown_connections 2024-09-06T14:24:20.210 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:20.213+0000 7fbd69bd1640 1 --2- 172.21.15.142:0/455302719 >> v2:172.21.15.142:6800/853729507 conn(0x7fbd50077530 0x7fbd500799f0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:20.211 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:20.213+0000 7fbd69bd1640 1 --2- 172.21.15.142:0/455302719 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbd64107d70 0x7fbd6419cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:20.211 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:20.213+0000 7fbd69bd1640 1 --2- 172.21.15.142:0/455302719 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbd641073f0 0x7fbd64197b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:20.211 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:20.213+0000 7fbd69bd1640 1 --2- 172.21.15.142:0/455302719 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbd641019e0 0x7fbd64197650 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:20.211 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:20.213+0000 7fbd69bd1640 1 -- 172.21.15.142:0/455302719 >> 172.21.15.142:0/455302719 conn(0x7fbd640fd450 msgr2=0x7fbd640ff840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:24:20.211 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:20.213+0000 7fbd69bd1640 1 -- 172.21.15.142:0/455302719 shutdown_connections 2024-09-06T14:24:20.211 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:20.213+0000 7fbd69bd1640 1 -- 172.21.15.142:0/455302719 wait complete. 2024-09-06T14:24:20.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:19 smithi162 ceph-mon[33787]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-09-06T14:24:20.921 INFO:tasks.cephadm:Adding ceph.rgw.foo.a on smithi142 2024-09-06T14:24:20.921 DEBUG:teuthology.orchestra.run.smithi162:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph orch apply rgw foo.a --placement '1;smithi142=foo.a' 2024-09-06T14:24:21.180 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.b/config 2024-09-06T14:24:21.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:20 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/455302719' entity='client.admin' cmd={"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} : dispatch 2024-09-06T14:24:21.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:20 smithi142 ceph-mon[37372]: from='client.? 172.21.15.142:0/455302719' entity='client.admin' cmd={"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} : dispatch 2024-09-06T14:24:21.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:20 smithi162 ceph-mon[33787]: from='client.? 172.21.15.142:0/455302719' entity='client.admin' cmd={"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} : dispatch 2024-09-06T14:24:21.767 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:21.763+0000 7fad456a2640 1 -- 172.21.15.162:0/3173352638 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fad401027a0 msgr2=0x7fad40102c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:21.767 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:21.763+0000 7fad456a2640 1 --2- 172.21.15.162:0/3173352638 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fad401027a0 0x7fad40102c20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fad28009930 tx=0x7fad2802a420 comp rx=0 tx=0).stop 2024-09-06T14:24:21.767 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:21.764+0000 7fad456a2640 1 -- 172.21.15.162:0/3173352638 shutdown_connections 2024-09-06T14:24:21.767 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:21.764+0000 7fad456a2640 1 --2- 172.21.15.162:0/3173352638 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fad40103160 0x7fad40109ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:21.767 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:21.764+0000 7fad456a2640 1 --2- 172.21.15.162:0/3173352638 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fad401027a0 0x7fad40102c20 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:21.767 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:21.764+0000 7fad456a2640 1 --2- 172.21.15.162:0/3173352638 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fad40101640 0x7fad40101a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:21.767 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:21.764+0000 7fad456a2640 1 -- 172.21.15.162:0/3173352638 >> 172.21.15.162:0/3173352638 conn(0x7fad400fcf10 msgr2=0x7fad400ff350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:24:21.767 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:21.764+0000 7fad456a2640 1 -- 172.21.15.162:0/3173352638 shutdown_connections 2024-09-06T14:24:21.767 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:21.764+0000 7fad456a2640 1 -- 172.21.15.162:0/3173352638 wait complete. 2024-09-06T14:24:21.768 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:21.764+0000 7fad456a2640 1 Processor -- start 2024-09-06T14:24:21.768 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:21.764+0000 7fad456a2640 1 -- start start 2024-09-06T14:24:21.768 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:21.765+0000 7fad456a2640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fad40101640 0x7fad4006b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:21.768 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:21.765+0000 7fad456a2640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fad401027a0 0x7fad4019cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:21.768 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:21.765+0000 7fad456a2640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fad40103160 0x7fad4019f540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:21.768 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:21.765+0000 7fad456a2640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fad4010c950 con 0x7fad40101640 2024-09-06T14:24:21.768 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:21.765+0000 7fad456a2640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fad4010c7d0 con 0x7fad401027a0 2024-09-06T14:24:21.768 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:21.765+0000 7fad456a2640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7fad4010cad0 con 0x7fad40103160 2024-09-06T14:24:21.768 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:21.765+0000 7fad367fc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fad401027a0 0x7fad4019cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:21.768 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:21.765+0000 7fad367fc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fad401027a0 0x7fad4019cf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:46584/0 (socket says 172.21.15.162:46584) 2024-09-06T14:24:21.768 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:21.765+0000 7fad367fc640 1 -- 172.21.15.162:0/967326170 learned_addr learned my addr 172.21.15.162:0/967326170 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-09-06T14:24:21.768 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:21.765+0000 7fad3effd640 1 --2- 172.21.15.162:0/967326170 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fad40101640 0x7fad4006b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:21.768 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:21.765+0000 7fad3f7fe640 1 --2- 172.21.15.162:0/967326170 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fad40103160 0x7fad4019f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:21.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:21.765+0000 7fad367fc640 1 -- 172.21.15.162:0/967326170 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fad40103160 msgr2=0x7fad4019f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:21.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:21.765+0000 7fad367fc640 1 --2- 172.21.15.162:0/967326170 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fad40103160 0x7fad4019f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:21.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:21.765+0000 7fad367fc640 1 -- 172.21.15.162:0/967326170 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fad40101640 msgr2=0x7fad4006b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:21.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:21.765+0000 7fad367fc640 1 --2- 172.21.15.162:0/967326170 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fad40101640 0x7fad4006b120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:21.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:21.765+0000 7fad367fc640 1 -- 172.21.15.162:0/967326170 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad4019fe00 con 0x7fad401027a0 2024-09-06T14:24:21.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:21.765+0000 7fad367fc640 1 --2- 172.21.15.162:0/967326170 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fad401027a0 0x7fad4019cf50 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fad28005940 tx=0x7fad28032c40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:24:21.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:21.765+0000 7fad3cff9640 1 -- 172.21.15.162:0/967326170 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad2803f030 con 0x7fad401027a0 2024-09-06T14:24:21.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:21.765+0000 7fad3cff9640 1 -- 172.21.15.162:0/967326170 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7fad28032e30 con 0x7fad401027a0 2024-09-06T14:24:21.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:21.765+0000 7fad456a2640 1 -- 172.21.15.162:0/967326170 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad401a0090 con 0x7fad401027a0 2024-09-06T14:24:21.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:21.765+0000 7fad3cff9640 1 -- 172.21.15.162:0/967326170 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad28035070 con 0x7fad401027a0 2024-09-06T14:24:21.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:21.765+0000 7fad456a2640 1 -- 172.21.15.162:0/967326170 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad401a05c0 con 0x7fad401027a0 2024-09-06T14:24:21.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:21.766+0000 7fad456a2640 1 -- 172.21.15.162:0/967326170 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad40101a40 con 0x7fad401027a0 2024-09-06T14:24:21.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:21.767+0000 7fad3cff9640 1 -- 172.21.15.162:0/967326170 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 15) ==== 99786+0+0 (secure 0 0 0) 0x7fad28042030 con 0x7fad401027a0 2024-09-06T14:24:21.770 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:21.768+0000 7fad3cff9640 1 --2- 172.21.15.162:0/967326170 >> v2:172.21.15.142:6800/853729507 conn(0x7fad2c077530 0x7fad2c0799f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:21.770 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:21.768+0000 7fad3cff9640 1 -- 172.21.15.162:0/967326170 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 3995+0+0 (secure 0 0 0) 0x7fad280b9510 con 0x7fad401027a0 2024-09-06T14:24:21.770 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:21.768+0000 7fad3effd640 1 --2- 172.21.15.162:0/967326170 >> v2:172.21.15.142:6800/853729507 conn(0x7fad2c077530 0x7fad2c0799f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:21.770 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:21.768+0000 7fad3effd640 1 --2- 172.21.15.162:0/967326170 >> v2:172.21.15.142:6800/853729507 conn(0x7fad2c077530 0x7fad2c0799f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fad3000bbe0 tx=0x7fad3000b390 comp rx=0 tx=0).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:24:21.770 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:21.772+0000 7fad3cff9640 1 -- 172.21.15.162:0/967326170 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7fad28082860 con 0x7fad401027a0 2024-09-06T14:24:21.931 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:21.935+0000 7fad456a2640 1 -- 172.21.15.162:0/967326170 --> v2:172.21.15.142:6800/853729507 -- mgr_command(tid 0: {"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi142=foo.a", "target": ["mon-mgr", ""]}) -- 0x7fad400632f0 con 0x7fad2c077530 2024-09-06T14:24:21.932 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:21 smithi162 ceph-mon[33787]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-09-06T14:24:21.940 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:21.944+0000 7fad3cff9640 1 -- 172.21.15.162:0/967326170 <== mgr.14146 v2:172.21.15.142:6800/853729507 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+30 (secure 0 0 0) 0x7fad400632f0 con 0x7fad2c077530 2024-09-06T14:24:21.940 INFO:teuthology.orchestra.run.smithi162.stdout:Scheduled rgw.foo.a update... 2024-09-06T14:24:21.942 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:21.946+0000 7fad456a2640 1 -- 172.21.15.162:0/967326170 >> v2:172.21.15.142:6800/853729507 conn(0x7fad2c077530 msgr2=0x7fad2c0799f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:21.942 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:21.946+0000 7fad456a2640 1 --2- 172.21.15.162:0/967326170 >> v2:172.21.15.142:6800/853729507 conn(0x7fad2c077530 0x7fad2c0799f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fad3000bbe0 tx=0x7fad3000b390 comp rx=0 tx=0).stop 2024-09-06T14:24:21.942 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:21.946+0000 7fad456a2640 1 -- 172.21.15.162:0/967326170 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fad401027a0 msgr2=0x7fad4019cf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:21.942 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:21.946+0000 7fad456a2640 1 --2- 172.21.15.162:0/967326170 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fad401027a0 0x7fad4019cf50 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fad28005940 tx=0x7fad28032c40 comp rx=0 tx=0).stop 2024-09-06T14:24:21.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:21.946+0000 7fad456a2640 1 -- 172.21.15.162:0/967326170 shutdown_connections 2024-09-06T14:24:21.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:21.946+0000 7fad456a2640 1 --2- 172.21.15.162:0/967326170 >> v2:172.21.15.142:6800/853729507 conn(0x7fad2c077530 0x7fad2c0799f0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:21.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:21.946+0000 7fad456a2640 1 --2- 172.21.15.162:0/967326170 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fad40103160 0x7fad4019f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:21.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:21.946+0000 7fad456a2640 1 --2- 172.21.15.162:0/967326170 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fad401027a0 0x7fad4019cf50 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:21.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:21.946+0000 7fad456a2640 1 --2- 172.21.15.162:0/967326170 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fad40101640 0x7fad4006b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:21.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:21.946+0000 7fad456a2640 1 -- 172.21.15.162:0/967326170 >> 172.21.15.162:0/967326170 conn(0x7fad400fcf10 msgr2=0x7fad400fea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:24:21.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:21.946+0000 7fad456a2640 1 -- 172.21.15.162:0/967326170 shutdown_connections 2024-09-06T14:24:21.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:21.946+0000 7fad456a2640 1 -- 172.21.15.162:0/967326170 wait complete. 2024-09-06T14:24:22.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:21 smithi142 ceph-mon[29958]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-09-06T14:24:22.208 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:21 smithi142 ceph-mon[37372]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-09-06T14:24:22.624 DEBUG:teuthology.orchestra.run.smithi142:rgw.foo.a> sudo journalctl -f -n 0 -u ceph-ce32462e-6c5a-11ef-bce4-c7b262605968@rgw.foo.a.service 2024-09-06T14:24:22.627 INFO:tasks.cephadm:Adding ceph.iscsi.iscsi.a on smithi162 2024-09-06T14:24:22.627 DEBUG:teuthology.orchestra.run.smithi162:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph osd pool create datapool 3 3 replicated 2024-09-06T14:24:22.871 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.b/config 2024-09-06T14:24:23.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:22 smithi142 ceph-mon[29958]: from='client.24313 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi142=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-06T14:24:23.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:22 smithi142 ceph-mon[29958]: Saving service rgw.foo.a spec with placement smithi142=foo.a;count:1 2024-09-06T14:24:23.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:22 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:23.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:22 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-06T14:24:23.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:22 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:24:23.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:22 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-06T14:24:23.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:22 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:23.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:22 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-06T14:24:23.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:22 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-06T14:24:23.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:22 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:23.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:22 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:24:23.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:22 smithi142 ceph-mon[29958]: Deploying daemon rgw.foo.a on smithi142 2024-09-06T14:24:23.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:22 smithi142 ceph-mon[37372]: from='client.24313 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi142=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-06T14:24:23.210 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:22 smithi142 ceph-mon[37372]: Saving service rgw.foo.a spec with placement smithi142=foo.a;count:1 2024-09-06T14:24:23.210 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:22 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:23.210 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:22 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-06T14:24:23.210 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:22 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:24:23.210 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:22 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-06T14:24:23.210 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:22 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:23.210 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:22 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-06T14:24:23.210 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:22 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-06T14:24:23.210 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:22 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:23.210 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:22 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:24:23.210 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:22 smithi142 ceph-mon[37372]: Deploying daemon rgw.foo.a on smithi142 2024-09-06T14:24:23.266 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:22 smithi162 ceph-mon[33787]: from='client.24313 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi142=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-06T14:24:23.266 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:22 smithi162 ceph-mon[33787]: Saving service rgw.foo.a spec with placement smithi142=foo.a;count:1 2024-09-06T14:24:23.267 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:22 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:23.267 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:22 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-06T14:24:23.267 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:22 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:24:23.267 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:22 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-06T14:24:23.267 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:22 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:23.267 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:22 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-06T14:24:23.267 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:22 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-06T14:24:23.267 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:22 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:23.267 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:22 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:24:23.267 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:22 smithi162 ceph-mon[33787]: Deploying daemon rgw.foo.a on smithi142 2024-09-06T14:24:23.353 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:23.355+0000 7f7cb4e15640 1 -- 172.21.15.162:0/1930371253 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7cb01038c0 msgr2=0x7f7cb0103d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:23.353 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:23.355+0000 7f7cb4e15640 1 --2- 172.21.15.162:0/1930371253 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7cb01038c0 0x7f7cb0103d40 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7ca4009930 tx=0x7f7ca402a420 comp rx=0 tx=0).stop 2024-09-06T14:24:23.353 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:23.355+0000 7f7cb4e15640 1 -- 172.21.15.162:0/1930371253 shutdown_connections 2024-09-06T14:24:23.353 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:23.355+0000 7f7cb4e15640 1 --2- 172.21.15.162:0/1930371253 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7cb0104280 0x7f7cb010ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:23.353 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:23.355+0000 7f7cb4e15640 1 --2- 172.21.15.162:0/1930371253 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7cb01038c0 0x7f7cb0103d40 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:23.353 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:23.355+0000 7f7cb4e15640 1 --2- 172.21.15.162:0/1930371253 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7cb0102760 0x7f7cb0102b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:23.353 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:23.355+0000 7f7cb4e15640 1 -- 172.21.15.162:0/1930371253 >> 172.21.15.162:0/1930371253 conn(0x7f7cb00fe050 msgr2=0x7f7cb0100470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:24:23.353 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:23.355+0000 7f7cb4e15640 1 -- 172.21.15.162:0/1930371253 shutdown_connections 2024-09-06T14:24:23.354 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:23.355+0000 7f7cb4e15640 1 -- 172.21.15.162:0/1930371253 wait complete. 2024-09-06T14:24:23.354 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:23.355+0000 7f7cb4e15640 1 Processor -- start 2024-09-06T14:24:23.354 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:23.356+0000 7f7cb4e15640 1 -- start start 2024-09-06T14:24:23.354 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:23.356+0000 7f7cb4e15640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7cb0102760 0x7f7cb0198220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:23.354 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:23.356+0000 7f7cb4e15640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7cb01038c0 0x7f7cb0198870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:23.354 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:23.356+0000 7f7cb4e15640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7cb0104280 0x7f7cb01994b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:23.354 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:23.356+0000 7f7cb4e15640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f7cb010daa0 con 0x7f7cb0104280 2024-09-06T14:24:23.354 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:23.356+0000 7f7cb4e15640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7cb010d920 con 0x7f7cb01038c0 2024-09-06T14:24:23.354 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:23.356+0000 7f7cb4e15640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7f7cb010dc20 con 0x7f7cb0102760 2024-09-06T14:24:23.354 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:23.356+0000 7f7cadd9b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7cb01038c0 0x7f7cb0198870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:23.354 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:23.356+0000 7f7cae59c640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7cb0102760 0x7f7cb0198220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:23.354 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:23.356+0000 7f7cae59c640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7cb0102760 0x7f7cb0198220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.162:55212/0 (socket says 172.21.15.162:55212) 2024-09-06T14:24:23.354 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:23.356+0000 7f7cadd9b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7cb01038c0 0x7f7cb0198870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:60084/0 (socket says 172.21.15.162:60084) 2024-09-06T14:24:23.354 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:23.356+0000 7f7cae59c640 1 -- 172.21.15.162:0/2293264195 learned_addr learned my addr 172.21.15.162:0/2293264195 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-09-06T14:24:23.355 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:23.356+0000 7f7caed9d640 1 --2- 172.21.15.162:0/2293264195 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7cb0104280 0x7f7cb01994b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:23.355 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:23.356+0000 7f7cae59c640 1 -- 172.21.15.162:0/2293264195 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7cb01038c0 msgr2=0x7f7cb0198870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:23.355 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:23.356+0000 7f7cae59c640 1 --2- 172.21.15.162:0/2293264195 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7cb01038c0 0x7f7cb0198870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:23.355 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:23.356+0000 7f7cae59c640 1 -- 172.21.15.162:0/2293264195 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7cb0104280 msgr2=0x7f7cb01994b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:23.355 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:23.356+0000 7f7cae59c640 1 --2- 172.21.15.162:0/2293264195 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7cb0104280 0x7f7cb01994b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:23.355 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:23.356+0000 7f7cae59c640 1 -- 172.21.15.162:0/2293264195 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cb0199d50 con 0x7f7cb0102760 2024-09-06T14:24:23.355 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:23.356+0000 7f7cae59c640 1 --2- 172.21.15.162:0/2293264195 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7cb0102760 0x7f7cb0198220 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f7ca000e9f0 tx=0x7f7ca000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:24:23.355 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:23.357+0000 7f7c9f7fe640 1 -- 172.21.15.162:0/2293264195 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ca000cd80 con 0x7f7cb0102760 2024-09-06T14:24:23.355 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:23.357+0000 7f7c9f7fe640 1 -- 172.21.15.162:0/2293264195 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7f7ca00108b0 con 0x7f7cb0102760 2024-09-06T14:24:23.355 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:23.357+0000 7f7cb4e15640 1 -- 172.21.15.162:0/2293264195 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7cb019a040 con 0x7f7cb0102760 2024-09-06T14:24:23.355 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:23.357+0000 7f7c9f7fe640 1 -- 172.21.15.162:0/2293264195 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ca0018870 con 0x7f7cb0102760 2024-09-06T14:24:23.355 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:23.357+0000 7f7cb4e15640 1 -- 172.21.15.162:0/2293264195 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7cb01a0720 con 0x7f7cb0102760 2024-09-06T14:24:23.356 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:23.358+0000 7f7cb4e15640 1 -- 172.21.15.162:0/2293264195 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7cb0102b60 con 0x7f7cb0102760 2024-09-06T14:24:23.356 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:23.358+0000 7f7c9f7fe640 1 -- 172.21.15.162:0/2293264195 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 15) ==== 99786+0+0 (secure 0 0 0) 0x7f7ca0010430 con 0x7f7cb0102760 2024-09-06T14:24:23.356 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:23.359+0000 7f7c9f7fe640 1 --2- 172.21.15.162:0/2293264195 >> v2:172.21.15.142:6800/853729507 conn(0x7f7c98077530 0x7f7c980799f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:23.357 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:23.359+0000 7f7c9f7fe640 1 -- 172.21.15.162:0/2293264195 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(50..50 src has 1..50) ==== 3995+0+0 (secure 0 0 0) 0x7f7ca00993f0 con 0x7f7cb0102760 2024-09-06T14:24:23.357 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:23.359+0000 7f7cadd9b640 1 --2- 172.21.15.162:0/2293264195 >> v2:172.21.15.142:6800/853729507 conn(0x7f7c98077530 0x7f7c980799f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:23.357 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:23.359+0000 7f7cadd9b640 1 --2- 172.21.15.162:0/2293264195 >> v2:172.21.15.142:6800/853729507 conn(0x7f7c98077530 0x7f7c980799f0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f7ca4009900 tx=0x7f7ca4000bc0 comp rx=0 tx=0).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:24:23.359 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:23.363+0000 7f7c9f7fe640 1 -- 172.21.15.162:0/2293264195 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7f7ca00626e0 con 0x7f7cb0102760 2024-09-06T14:24:23.511 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:23.514+0000 7f7cb4e15640 1 -- 172.21.15.162:0/2293264195 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"} v 0) -- 0x7f7cb0109300 con 0x7f7cb0102760 2024-09-06T14:24:23.959 INFO:journalctl@ceph.rgw.foo.a.smithi142.stdout:Sep 06 14:24:23 smithi142 systemd[1]: Starting Ceph rgw.foo.a for ce32462e-6c5a-11ef-bce4-c7b262605968... 2024-09-06T14:24:23.973 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:23.977+0000 7f7c9f7fe640 1 -- 172.21.15.162:0/2293264195 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]=0 pool 'datapool' created v51) ==== 160+0+0 (secure 0 0 0) 0x7f7ca0066ea0 con 0x7f7cb0102760 2024-09-06T14:24:23.973 INFO:teuthology.orchestra.run.smithi162.stderr:pool 'datapool' created 2024-09-06T14:24:23.976 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:23.979+0000 7f7cb4e15640 1 -- 172.21.15.162:0/2293264195 >> v2:172.21.15.142:6800/853729507 conn(0x7f7c98077530 msgr2=0x7f7c980799f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:23.976 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:23.979+0000 7f7cb4e15640 1 --2- 172.21.15.162:0/2293264195 >> v2:172.21.15.142:6800/853729507 conn(0x7f7c98077530 0x7f7c980799f0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f7ca4009900 tx=0x7f7ca4000bc0 comp rx=0 tx=0).stop 2024-09-06T14:24:23.976 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:23.979+0000 7f7cb4e15640 1 -- 172.21.15.162:0/2293264195 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7cb0102760 msgr2=0x7f7cb0198220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:23.976 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:23.979+0000 7f7cb4e15640 1 --2- 172.21.15.162:0/2293264195 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7cb0102760 0x7f7cb0198220 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f7ca000e9f0 tx=0x7f7ca000eeb0 comp rx=0 tx=0).stop 2024-09-06T14:24:23.976 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:23.979+0000 7f7cb4e15640 1 -- 172.21.15.162:0/2293264195 shutdown_connections 2024-09-06T14:24:23.976 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:23.979+0000 7f7cb4e15640 1 --2- 172.21.15.162:0/2293264195 >> v2:172.21.15.142:6800/853729507 conn(0x7f7c98077530 0x7f7c980799f0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:23.976 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:23.979+0000 7f7cb4e15640 1 --2- 172.21.15.162:0/2293264195 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7cb0104280 0x7f7cb01994b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:23.976 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:23.980+0000 7f7cb4e15640 1 --2- 172.21.15.162:0/2293264195 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7cb01038c0 0x7f7cb0198870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:23.976 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:23.980+0000 7f7cb4e15640 1 --2- 172.21.15.162:0/2293264195 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7cb0102760 0x7f7cb0198220 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:23.976 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:23.980+0000 7f7cb4e15640 1 -- 172.21.15.162:0/2293264195 >> 172.21.15.162:0/2293264195 conn(0x7f7cb00fe050 msgr2=0x7f7cb00ffc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:24:23.976 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:23.980+0000 7f7cb4e15640 1 -- 172.21.15.162:0/2293264195 shutdown_connections 2024-09-06T14:24:23.977 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:23.980+0000 7f7cb4e15640 1 -- 172.21.15.162:0/2293264195 wait complete. 2024-09-06T14:24:24.268 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:23 smithi142 ceph-mon[29958]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-09-06T14:24:24.269 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:23 smithi142 ceph-mon[29958]: from='client.? 172.21.15.162:0/2293264195' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"} : dispatch 2024-09-06T14:24:24.269 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:23 smithi142 ceph-mon[29958]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"} : dispatch 2024-09-06T14:24:24.269 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:23 smithi142 ceph-mon[37372]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-09-06T14:24:24.269 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:23 smithi142 ceph-mon[37372]: from='client.? 172.21.15.162:0/2293264195' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"} : dispatch 2024-09-06T14:24:24.269 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:23 smithi142 ceph-mon[37372]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"} : dispatch 2024-09-06T14:24:24.269 INFO:journalctl@ceph.rgw.foo.a.smithi142.stdout:Sep 06 14:24:24 smithi142 podman[63238]: 2024-09-06 14:24:24.041873304 +0000 UTC m=+0.015687527 image pull 1c45aca0610a312233aa2613f7b416b2884ae14779f1297041485ab28f1c8110 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c86edd48571ab22373283621eb9feebce9df9c291d4164cfc4df1a09493e1f30 2024-09-06T14:24:24.269 INFO:journalctl@ceph.rgw.foo.a.smithi142.stdout:Sep 06 14:24:24 smithi142 podman[63238]: 2024-09-06 14:24:24.155550404 +0000 UTC m=+0.129364663 container create 47d61cae342f69fa2cb6b26d9e0422c4fadd727043e2c381234c1c3e711b6b0c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c86edd48571ab22373283621eb9feebce9df9c291d4164cfc4df1a09493e1f30, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-rgw-foo-a, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-athakkar-testing-2024-09-04-1837-8b38c33, org.label-schema.build-date=20240731, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS) 2024-09-06T14:24:24.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:23 smithi162 ceph-mon[33787]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-09-06T14:24:24.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:23 smithi162 ceph-mon[33787]: from='client.? 172.21.15.162:0/2293264195' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"} : dispatch 2024-09-06T14:24:24.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:23 smithi162 ceph-mon[33787]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"} : dispatch 2024-09-06T14:24:24.705 DEBUG:teuthology.orchestra.run.smithi162:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- rbd pool init datapool 2024-09-06T14:24:24.708 INFO:journalctl@ceph.rgw.foo.a.smithi142.stdout:Sep 06 14:24:24 smithi142 podman[63238]: 2024-09-06 14:24:24.383104083 +0000 UTC m=+0.356918297 container init 47d61cae342f69fa2cb6b26d9e0422c4fadd727043e2c381234c1c3e711b6b0c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c86edd48571ab22373283621eb9feebce9df9c291d4164cfc4df1a09493e1f30, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-rgw-foo-a, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, RELEASE=wip-athakkar-testing-2024-09-04-1837-8b38c33, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731) 2024-09-06T14:24:24.708 INFO:journalctl@ceph.rgw.foo.a.smithi142.stdout:Sep 06 14:24:24 smithi142 podman[63238]: 2024-09-06 14:24:24.38593119 +0000 UTC m=+0.359745413 container start 47d61cae342f69fa2cb6b26d9e0422c4fadd727043e2c381234c1c3e711b6b0c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c86edd48571ab22373283621eb9feebce9df9c291d4164cfc4df1a09493e1f30, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-rgw-foo-a, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, RELEASE=wip-athakkar-testing-2024-09-04-1837-8b38c33, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-09-06T14:24:24.708 INFO:journalctl@ceph.rgw.foo.a.smithi142.stdout:Sep 06 14:24:24 smithi142 bash[63238]: 47d61cae342f69fa2cb6b26d9e0422c4fadd727043e2c381234c1c3e711b6b0c 2024-09-06T14:24:24.708 INFO:journalctl@ceph.rgw.foo.a.smithi142.stdout:Sep 06 14:24:24 smithi142 systemd[1]: Started Ceph rgw.foo.a for ce32462e-6c5a-11ef-bce4-c7b262605968. 2024-09-06T14:24:24.949 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.b/config 2024-09-06T14:24:25.236 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:24 smithi162 ceph-mon[33787]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-09-06T14:24:25.237 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:24 smithi162 ceph-mon[33787]: osdmap e51: 8 total, 8 up, 8 in 2024-09-06T14:24:25.237 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:24 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:25.237 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:24 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:25.237 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:24 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:25.237 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:24 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:25.237 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:24 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:25.237 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:24 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-06T14:24:25.287 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:24 smithi142 ceph-mon[29958]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-09-06T14:24:25.287 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:24 smithi142 ceph-mon[29958]: osdmap e51: 8 total, 8 up, 8 in 2024-09-06T14:24:25.287 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:24 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:25.288 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:24 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:25.288 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:24 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:25.288 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:24 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:25.288 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:24 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:25.288 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:24 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-06T14:24:25.288 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:24 smithi142 ceph-mon[37372]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-09-06T14:24:25.288 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:24 smithi142 ceph-mon[37372]: osdmap e51: 8 total, 8 up, 8 in 2024-09-06T14:24:25.288 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:24 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:25.288 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:24 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:25.288 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:24 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:25.288 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:24 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:25.289 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:24 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:25.289 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:24 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-06T14:24:26.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:25 smithi142 ceph-mon[37372]: Saving service rgw.foo.a spec with placement smithi142=foo.a;count:1 2024-09-06T14:24:26.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:25 smithi142 ceph-mon[37372]: pgmap v174: 4 pgs: 3 unknown, 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-09-06T14:24:26.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:25 smithi142 ceph-mon[37372]: osdmap e52: 8 total, 8 up, 8 in 2024-09-06T14:24:26.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:25 smithi142 ceph-mon[37372]: from='client.? 172.21.15.142:0/691380340' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"} : dispatch 2024-09-06T14:24:26.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:25 smithi142 ceph-mon[37372]: from='client.? 172.21.15.162:0/597721505' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "datapool","app": "rbd"} : dispatch 2024-09-06T14:24:26.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:25 smithi142 ceph-mon[37372]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "datapool","app": "rbd"} : dispatch 2024-09-06T14:24:26.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:25 smithi142 ceph-mon[37372]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-06T14:24:26.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:25 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:26.210 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:25 smithi142 ceph-mon[29958]: Saving service rgw.foo.a spec with placement smithi142=foo.a;count:1 2024-09-06T14:24:26.210 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:25 smithi142 ceph-mon[29958]: pgmap v174: 4 pgs: 3 unknown, 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-09-06T14:24:26.210 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:25 smithi142 ceph-mon[29958]: osdmap e52: 8 total, 8 up, 8 in 2024-09-06T14:24:26.210 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:25 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/691380340' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"} : dispatch 2024-09-06T14:24:26.210 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:25 smithi142 ceph-mon[29958]: from='client.? 172.21.15.162:0/597721505' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "datapool","app": "rbd"} : dispatch 2024-09-06T14:24:26.210 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:25 smithi142 ceph-mon[29958]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "datapool","app": "rbd"} : dispatch 2024-09-06T14:24:26.210 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:25 smithi142 ceph-mon[29958]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-06T14:24:26.210 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:25 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:26.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:25 smithi162 ceph-mon[33787]: Saving service rgw.foo.a spec with placement smithi142=foo.a;count:1 2024-09-06T14:24:26.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:25 smithi162 ceph-mon[33787]: pgmap v174: 4 pgs: 3 unknown, 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-09-06T14:24:26.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:25 smithi162 ceph-mon[33787]: osdmap e52: 8 total, 8 up, 8 in 2024-09-06T14:24:26.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:25 smithi162 ceph-mon[33787]: from='client.? 172.21.15.142:0/691380340' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"} : dispatch 2024-09-06T14:24:26.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:25 smithi162 ceph-mon[33787]: from='client.? 172.21.15.162:0/597721505' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "datapool","app": "rbd"} : dispatch 2024-09-06T14:24:26.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:25 smithi162 ceph-mon[33787]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "datapool","app": "rbd"} : dispatch 2024-09-06T14:24:26.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:25 smithi162 ceph-mon[33787]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-06T14:24:26.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:25 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:27.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:26 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:27.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:26 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:24:27.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:26 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-06T14:24:27.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:26 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/691380340' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-09-06T14:24:27.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:26 smithi142 ceph-mon[29958]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-09-06T14:24:27.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:26 smithi142 ceph-mon[29958]: osdmap e53: 8 total, 8 up, 8 in 2024-09-06T14:24:27.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:26 smithi142 ceph-mon[29958]: Checking dashboard <-> RGW credentials 2024-09-06T14:24:27.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:26 smithi142 ceph-mon[29958]: pgmap v177: 36 pgs: 12 creating+peering, 12 unknown, 12 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 511 B/s wr, 2 op/s 2024-09-06T14:24:27.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:26 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:27.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:26 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:24:27.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:26 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-06T14:24:27.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:26 smithi142 ceph-mon[37372]: from='client.? 172.21.15.142:0/691380340' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-09-06T14:24:27.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:26 smithi142 ceph-mon[37372]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-09-06T14:24:27.210 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:26 smithi142 ceph-mon[37372]: osdmap e53: 8 total, 8 up, 8 in 2024-09-06T14:24:27.210 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:26 smithi142 ceph-mon[37372]: Checking dashboard <-> RGW credentials 2024-09-06T14:24:27.210 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:26 smithi142 ceph-mon[37372]: pgmap v177: 36 pgs: 12 creating+peering, 12 unknown, 12 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 511 B/s wr, 2 op/s 2024-09-06T14:24:27.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:26 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:27.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:26 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:24:27.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:26 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-06T14:24:27.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:26 smithi162 ceph-mon[33787]: from='client.? 172.21.15.142:0/691380340' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-09-06T14:24:27.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:26 smithi162 ceph-mon[33787]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-09-06T14:24:27.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:26 smithi162 ceph-mon[33787]: osdmap e53: 8 total, 8 up, 8 in 2024-09-06T14:24:27.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:26 smithi162 ceph-mon[33787]: Checking dashboard <-> RGW credentials 2024-09-06T14:24:27.298 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:26 smithi162 ceph-mon[33787]: pgmap v177: 36 pgs: 12 creating+peering, 12 unknown, 12 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 511 B/s wr, 2 op/s 2024-09-06T14:24:28.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:27 smithi162 ceph-mon[33787]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-06T14:24:28.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:27 smithi162 ceph-mon[33787]: osdmap e54: 8 total, 8 up, 8 in 2024-09-06T14:24:28.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:27 smithi162 ceph-mon[33787]: from='client.? 172.21.15.142:0/4111201236' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"} : dispatch 2024-09-06T14:24:28.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:27 smithi162 ceph-mon[33787]: from='client.? 172.21.15.142:0/3263656736' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"} : dispatch 2024-09-06T14:24:28.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:27 smithi162 ceph-mon[33787]: from='client.? ' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"} : dispatch 2024-09-06T14:24:28.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:27 smithi162 ceph-mon[33787]: from='client.? ' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"} : dispatch 2024-09-06T14:24:28.458 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:27 smithi142 ceph-mon[29958]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-06T14:24:28.459 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:27 smithi142 ceph-mon[29958]: osdmap e54: 8 total, 8 up, 8 in 2024-09-06T14:24:28.459 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:27 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/4111201236' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"} : dispatch 2024-09-06T14:24:28.459 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:27 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/3263656736' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"} : dispatch 2024-09-06T14:24:28.459 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:27 smithi142 ceph-mon[29958]: from='client.? ' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"} : dispatch 2024-09-06T14:24:28.459 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:27 smithi142 ceph-mon[29958]: from='client.? ' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"} : dispatch 2024-09-06T14:24:28.459 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:27 smithi142 ceph-mon[37372]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-06T14:24:28.459 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:27 smithi142 ceph-mon[37372]: osdmap e54: 8 total, 8 up, 8 in 2024-09-06T14:24:28.459 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:27 smithi142 ceph-mon[37372]: from='client.? 172.21.15.142:0/4111201236' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"} : dispatch 2024-09-06T14:24:28.459 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:27 smithi142 ceph-mon[37372]: from='client.? 172.21.15.142:0/3263656736' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"} : dispatch 2024-09-06T14:24:28.460 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:27 smithi142 ceph-mon[37372]: from='client.? ' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"} : dispatch 2024-09-06T14:24:28.460 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:27 smithi142 ceph-mon[37372]: from='client.? ' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"} : dispatch 2024-09-06T14:24:28.696 DEBUG:teuthology.orchestra.run.smithi162:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph orch apply iscsi datapool admin admin --trusted_ip_list 172.21.15.162 --placement '1;smithi162=iscsi.a' 2024-09-06T14:24:28.945 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.b/config 2024-09-06T14:24:29.252 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:29 smithi162 ceph-mon[33787]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-06T14:24:29.252 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:29 smithi162 ceph-mon[33787]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-06T14:24:29.253 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:29 smithi162 ceph-mon[33787]: osdmap e55: 8 total, 8 up, 8 in 2024-09-06T14:24:29.253 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:29 smithi162 ceph-mon[33787]: pgmap v180: 68 pgs: 24 creating+peering, 20 unknown, 24 active+clean; 578 KiB data, 216 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-09-06T14:24:29.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:29.446+0000 7f979faa9640 1 -- 172.21.15.162:0/2696474801 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f979806b3d0 msgr2=0x7f9798105360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:29.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:29.446+0000 7f979faa9640 1 --2- 172.21.15.162:0/2696474801 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f979806b3d0 0x7f9798105360 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9794009900 tx=0x7f979402a410 comp rx=0 tx=0).stop 2024-09-06T14:24:29.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:29.447+0000 7f979faa9640 1 -- 172.21.15.162:0/2696474801 shutdown_connections 2024-09-06T14:24:29.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:29.447+0000 7f979faa9640 1 --2- 172.21.15.162:0/2696474801 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9798108450 0x7f979810a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:29.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:29.447+0000 7f979faa9640 1 --2- 172.21.15.162:0/2696474801 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f97981058a0 0x7f9798107d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:29.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:29.447+0000 7f979faa9640 1 --2- 172.21.15.162:0/2696474801 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f979806b3d0 0x7f9798105360 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:29.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:29.447+0000 7f979faa9640 1 -- 172.21.15.162:0/2696474801 >> 172.21.15.162:0/2696474801 conn(0x7f97980fcf10 msgr2=0x7f97980ff350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:24:29.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:29.447+0000 7f979faa9640 1 -- 172.21.15.162:0/2696474801 shutdown_connections 2024-09-06T14:24:29.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:29.447+0000 7f979faa9640 1 -- 172.21.15.162:0/2696474801 wait complete. 2024-09-06T14:24:29.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:29.447+0000 7f979faa9640 1 Processor -- start 2024-09-06T14:24:29.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:29.447+0000 7f979faa9640 1 -- start start 2024-09-06T14:24:29.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:29.447+0000 7f979faa9640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f979806b3d0 0x7f97981981d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:29.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:29.447+0000 7f979faa9640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f97981058a0 0x7f9798198820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:29.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:29.447+0000 7f979faa9640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9798108450 0x7f9798199460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:29.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:29.447+0000 7f979faa9640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f979810da30 con 0x7f979806b3d0 2024-09-06T14:24:29.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:29.447+0000 7f979faa9640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f979810d8b0 con 0x7f9798108450 2024-09-06T14:24:29.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:29.447+0000 7f979faa9640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7f979810dbb0 con 0x7f97981058a0 2024-09-06T14:24:29.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:29.447+0000 7f979e046640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9798108450 0x7f9798199460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:29.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:29.448+0000 7f979e046640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9798108450 0x7f9798199460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:60120/0 (socket says 172.21.15.162:60120) 2024-09-06T14:24:29.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:29.448+0000 7f979e046640 1 -- 172.21.15.162:0/1145848193 learned_addr learned my addr 172.21.15.162:0/1145848193 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-09-06T14:24:29.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:29.448+0000 7f979d845640 1 --2- 172.21.15.162:0/1145848193 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f979806b3d0 0x7f97981981d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:29.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:29.448+0000 7f979d044640 1 --2- 172.21.15.162:0/1145848193 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f97981058a0 0x7f9798198820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:29.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:29.448+0000 7f979e046640 1 -- 172.21.15.162:0/1145848193 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f97981058a0 msgr2=0x7f9798198820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:29.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:29.448+0000 7f979e046640 1 --2- 172.21.15.162:0/1145848193 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f97981058a0 0x7f9798198820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:29.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:29.448+0000 7f979e046640 1 -- 172.21.15.162:0/1145848193 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f979806b3d0 msgr2=0x7f97981981d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:29.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:29.448+0000 7f979e046640 1 --2- 172.21.15.162:0/1145848193 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f979806b3d0 0x7f97981981d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:29.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:29.448+0000 7f979e046640 1 -- 172.21.15.162:0/1145848193 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9798199d00 con 0x7f9798108450 2024-09-06T14:24:29.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:29.448+0000 7f979e046640 1 --2- 172.21.15.162:0/1145848193 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9798108450 0x7f9798199460 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f979000eaa0 tx=0x7f979000ef60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:24:29.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:29.448+0000 7f978effd640 1 -- 172.21.15.162:0/1145848193 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f979000cdb0 con 0x7f9798108450 2024-09-06T14:24:29.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:29.448+0000 7f979faa9640 1 -- 172.21.15.162:0/1145848193 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9798199ff0 con 0x7f9798108450 2024-09-06T14:24:29.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:29.448+0000 7f978effd640 1 -- 172.21.15.162:0/1145848193 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7f9790005020 con 0x7f9798108450 2024-09-06T14:24:29.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:29.448+0000 7f978effd640 1 -- 172.21.15.162:0/1145848193 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97900116d0 con 0x7f9798108450 2024-09-06T14:24:29.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:29.448+0000 7f979faa9640 1 -- 172.21.15.162:0/1145848193 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97981a06d0 con 0x7f9798108450 2024-09-06T14:24:29.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:29.449+0000 7f979faa9640 1 -- 172.21.15.162:0/1145848193 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9798105d00 con 0x7f9798108450 2024-09-06T14:24:29.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:29.450+0000 7f978effd640 1 -- 172.21.15.162:0/1145848193 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 15) ==== 99786+0+0 (secure 0 0 0) 0x7f9790004a40 con 0x7f9798108450 2024-09-06T14:24:29.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:29.450+0000 7f978effd640 1 --2- 172.21.15.162:0/1145848193 >> v2:172.21.15.142:6800/853729507 conn(0x7f9784077530 0x7f97840799f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:29.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:29.450+0000 7f978effd640 1 -- 172.21.15.162:0/1145848193 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5463+0+0 (secure 0 0 0) 0x7f97900996b0 con 0x7f9798108450 2024-09-06T14:24:29.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:29.450+0000 7f979d845640 1 --2- 172.21.15.162:0/1145848193 >> v2:172.21.15.142:6800/853729507 conn(0x7f9784077530 0x7f97840799f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:29.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:29.451+0000 7f979d845640 1 --2- 172.21.15.162:0/1145848193 >> v2:172.21.15.142:6800/853729507 conn(0x7f9784077530 0x7f97840799f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f9794009230 tx=0x7f97940058d0 comp rx=0 tx=0).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:24:29.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:29.454+0000 7f978effd640 1 -- 172.21.15.162:0/1145848193 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7f9790016030 con 0x7f9798108450 2024-09-06T14:24:29.458 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:29 smithi142 ceph-mon[29958]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-06T14:24:29.458 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:29 smithi142 ceph-mon[29958]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-06T14:24:29.458 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:29 smithi142 ceph-mon[29958]: osdmap e55: 8 total, 8 up, 8 in 2024-09-06T14:24:29.458 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:29 smithi142 ceph-mon[29958]: pgmap v180: 68 pgs: 24 creating+peering, 20 unknown, 24 active+clean; 578 KiB data, 216 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-09-06T14:24:29.459 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:29 smithi142 ceph-mon[37372]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-06T14:24:29.459 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:29 smithi142 ceph-mon[37372]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-06T14:24:29.459 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:29 smithi142 ceph-mon[37372]: osdmap e55: 8 total, 8 up, 8 in 2024-09-06T14:24:29.459 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:29 smithi142 ceph-mon[37372]: pgmap v180: 68 pgs: 24 creating+peering, 20 unknown, 24 active+clean; 578 KiB data, 216 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-09-06T14:24:29.614 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:29.617+0000 7f979faa9640 1 -- 172.21.15.162:0/1145848193 --> v2:172.21.15.142:6800/853729507 -- mgr_command(tid 0: {"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.162", "placement": "1;smithi162=iscsi.a", "target": ["mon-mgr", ""]}) -- 0x7f979802e700 con 0x7f9784077530 2024-09-06T14:24:29.622 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:29.626+0000 7f978effd640 1 -- 172.21.15.162:0/1145848193 <== mgr.14146 v2:172.21.15.142:6800/853729507 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+35 (secure 0 0 0) 0x7f979802e700 con 0x7f9784077530 2024-09-06T14:24:29.623 INFO:teuthology.orchestra.run.smithi162.stdout:Scheduled iscsi.datapool update... 2024-09-06T14:24:29.625 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:29.628+0000 7f979faa9640 1 -- 172.21.15.162:0/1145848193 >> v2:172.21.15.142:6800/853729507 conn(0x7f9784077530 msgr2=0x7f97840799f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:29.625 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:29.628+0000 7f979faa9640 1 --2- 172.21.15.162:0/1145848193 >> v2:172.21.15.142:6800/853729507 conn(0x7f9784077530 0x7f97840799f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f9794009230 tx=0x7f97940058d0 comp rx=0 tx=0).stop 2024-09-06T14:24:29.625 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:29.628+0000 7f979faa9640 1 -- 172.21.15.162:0/1145848193 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9798108450 msgr2=0x7f9798199460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:29.625 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:29.628+0000 7f979faa9640 1 --2- 172.21.15.162:0/1145848193 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9798108450 0x7f9798199460 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f979000eaa0 tx=0x7f979000ef60 comp rx=0 tx=0).stop 2024-09-06T14:24:29.625 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:29.628+0000 7f979faa9640 1 -- 172.21.15.162:0/1145848193 shutdown_connections 2024-09-06T14:24:29.625 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:29.628+0000 7f979faa9640 1 --2- 172.21.15.162:0/1145848193 >> v2:172.21.15.142:6800/853729507 conn(0x7f9784077530 0x7f97840799f0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:29.625 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:29.628+0000 7f979faa9640 1 --2- 172.21.15.162:0/1145848193 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9798108450 0x7f9798199460 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:29.625 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:29.628+0000 7f979faa9640 1 --2- 172.21.15.162:0/1145848193 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f97981058a0 0x7f9798198820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:29.625 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:29.628+0000 7f979faa9640 1 --2- 172.21.15.162:0/1145848193 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f979806b3d0 0x7f97981981d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:29.625 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:29.628+0000 7f979faa9640 1 -- 172.21.15.162:0/1145848193 >> 172.21.15.162:0/1145848193 conn(0x7f97980fcf10 msgr2=0x7f9798103b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:24:29.625 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:29.628+0000 7f979faa9640 1 -- 172.21.15.162:0/1145848193 shutdown_connections 2024-09-06T14:24:29.625 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:29.628+0000 7f979faa9640 1 -- 172.21.15.162:0/1145848193 wait complete. 2024-09-06T14:24:30.200 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:30 smithi162 ceph-mon[33787]: osdmap e56: 8 total, 8 up, 8 in 2024-09-06T14:24:30.200 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:30 smithi162 ceph-mon[33787]: from='client.? 172.21.15.142:0/4111201236' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"} : dispatch 2024-09-06T14:24:30.200 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:30 smithi162 ceph-mon[33787]: from='client.? 172.21.15.142:0/3263656736' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"} : dispatch 2024-09-06T14:24:30.200 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:30 smithi162 ceph-mon[33787]: from='client.? ' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"} : dispatch 2024-09-06T14:24:30.200 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:30 smithi162 ceph-mon[33787]: from='client.? ' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"} : dispatch 2024-09-06T14:24:30.201 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:30 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:30.201 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:30 smithi162 ceph-mon[33787]: from='client.24367 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.162", "placement": "1;smithi162=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-06T14:24:30.201 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:30 smithi162 ceph-mon[33787]: Saving service iscsi.datapool spec with placement smithi162=iscsi.a;count:1 2024-09-06T14:24:30.201 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:30 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:30.226 INFO:tasks.cephadm:Distributing iscsi-gateway.cfg... 2024-09-06T14:24:30.227 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-09-06T14:24:30.227 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-09-06T14:24:30.255 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:30 smithi142 ceph-mon[37372]: osdmap e56: 8 total, 8 up, 8 in 2024-09-06T14:24:30.256 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:30 smithi142 ceph-mon[37372]: from='client.? 172.21.15.142:0/4111201236' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"} : dispatch 2024-09-06T14:24:30.256 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:30 smithi142 ceph-mon[37372]: from='client.? 172.21.15.142:0/3263656736' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"} : dispatch 2024-09-06T14:24:30.256 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:30 smithi142 ceph-mon[37372]: from='client.? ' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"} : dispatch 2024-09-06T14:24:30.256 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:30 smithi142 ceph-mon[37372]: from='client.? ' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"} : dispatch 2024-09-06T14:24:30.256 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:30 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:30.256 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:30 smithi142 ceph-mon[37372]: from='client.24367 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.162", "placement": "1;smithi162=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-06T14:24:30.256 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:30 smithi142 ceph-mon[37372]: Saving service iscsi.datapool spec with placement smithi162=iscsi.a;count:1 2024-09-06T14:24:30.256 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:30 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:30.256 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:30 smithi142 ceph-mon[29958]: osdmap e56: 8 total, 8 up, 8 in 2024-09-06T14:24:30.257 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:30 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/4111201236' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"} : dispatch 2024-09-06T14:24:30.257 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:30 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/3263656736' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"} : dispatch 2024-09-06T14:24:30.257 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:30 smithi142 ceph-mon[29958]: from='client.? ' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"} : dispatch 2024-09-06T14:24:30.257 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:30 smithi142 ceph-mon[29958]: from='client.? ' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"} : dispatch 2024-09-06T14:24:30.257 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:30 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:30.257 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:30 smithi142 ceph-mon[29958]: from='client.24367 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.162", "placement": "1;smithi162=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-06T14:24:30.257 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:30 smithi142 ceph-mon[29958]: Saving service iscsi.datapool spec with placement smithi162=iscsi.a;count:1 2024-09-06T14:24:30.257 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:30 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:30.258 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-09-06T14:24:30.258 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-09-06T14:24:30.284 DEBUG:teuthology.orchestra.run.smithi162:iscsi.iscsi.a> sudo journalctl -f -n 0 -u ceph-ce32462e-6c5a-11ef-bce4-c7b262605968@iscsi.iscsi.a.service 2024-09-06T14:24:30.326 INFO:tasks.cephadm:Adding prometheus.a on smithi162 2024-09-06T14:24:30.327 DEBUG:teuthology.orchestra.run.smithi162:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph orch apply prometheus '1;smithi162=a' 2024-09-06T14:24:30.609 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.b/config 2024-09-06T14:24:31.011 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:31 smithi162 ceph-mon[33787]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-06T14:24:31.011 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:31 smithi162 ceph-mon[33787]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-06T14:24:31.011 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:31 smithi162 ceph-mon[33787]: osdmap e57: 8 total, 8 up, 8 in 2024-09-06T14:24:31.012 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:31 smithi162 ceph-mon[33787]: pgmap v183: 100 pgs: 12 creating+peering, 23 unknown, 65 active+clean; 578 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-09-06T14:24:31.037 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:31.035+0000 7f780b17f640 1 -- 172.21.15.162:0/619948094 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f780406bdf0 msgr2=0x7f7804105350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:31.037 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:31.035+0000 7f780b17f640 1 --2- 172.21.15.162:0/619948094 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f780406bdf0 0x7f7804105350 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f77fc009900 tx=0x7f77fc02a410 comp rx=0 tx=0).stop 2024-09-06T14:24:31.037 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:31.035+0000 7f780b17f640 1 -- 172.21.15.162:0/619948094 shutdown_connections 2024-09-06T14:24:31.037 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:31.035+0000 7f780b17f640 1 --2- 172.21.15.162:0/619948094 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7804108490 0x7f780410a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:31.037 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:31.035+0000 7f780b17f640 1 --2- 172.21.15.162:0/619948094 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f78041058e0 0x7f7804107d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:31.037 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:31.035+0000 7f780b17f640 1 --2- 172.21.15.162:0/619948094 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f780406bdf0 0x7f7804105350 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:31.038 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:31.035+0000 7f780b17f640 1 -- 172.21.15.162:0/619948094 >> 172.21.15.162:0/619948094 conn(0x7f78040fcf60 msgr2=0x7f78040ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:24:31.038 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:31.035+0000 7f780b17f640 1 -- 172.21.15.162:0/619948094 shutdown_connections 2024-09-06T14:24:31.038 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:31.035+0000 7f780b17f640 1 -- 172.21.15.162:0/619948094 wait complete. 2024-09-06T14:24:31.038 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:31.035+0000 7f780b17f640 1 Processor -- start 2024-09-06T14:24:31.038 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:31.036+0000 7f780b17f640 1 -- start start 2024-09-06T14:24:31.038 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:31.036+0000 7f780b17f640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f780406bdf0 0x7f78041982d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:31.038 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:31.036+0000 7f780b17f640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f78041058e0 0x7f7804198920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:31.038 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:31.036+0000 7f780b17f640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7804108490 0x7f7804199560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:31.038 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:31.036+0000 7f780b17f640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f780410dac0 con 0x7f7804108490 2024-09-06T14:24:31.038 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:31.036+0000 7f780b17f640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f780410d940 con 0x7f780406bdf0 2024-09-06T14:24:31.038 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:31.036+0000 7f780b17f640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7f780410dc40 con 0x7f78041058e0 2024-09-06T14:24:31.038 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:31.036+0000 7f7808f1b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f780406bdf0 0x7f78041982d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:31.038 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:31.036+0000 7f7808f1b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f780406bdf0 0x7f78041982d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:60134/0 (socket says 172.21.15.162:60134) 2024-09-06T14:24:31.038 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:31.036+0000 7f7808f1b640 1 -- 172.21.15.162:0/3340327985 learned_addr learned my addr 172.21.15.162:0/3340327985 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-09-06T14:24:31.039 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:31.036+0000 7f7803fff640 1 --2- 172.21.15.162:0/3340327985 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f78041058e0 0x7f7804198920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:31.039 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:31.036+0000 7f780971c640 1 --2- 172.21.15.162:0/3340327985 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7804108490 0x7f7804199560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:31.039 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:31.036+0000 7f7808f1b640 1 -- 172.21.15.162:0/3340327985 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f78041058e0 msgr2=0x7f7804198920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:31.039 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:31.036+0000 7f7808f1b640 1 --2- 172.21.15.162:0/3340327985 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f78041058e0 0x7f7804198920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:31.039 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:31.036+0000 7f7808f1b640 1 -- 172.21.15.162:0/3340327985 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7804108490 msgr2=0x7f7804199560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:31.039 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:31.036+0000 7f7808f1b640 1 --2- 172.21.15.162:0/3340327985 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7804108490 0x7f7804199560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:31.039 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:31.036+0000 7f7808f1b640 1 -- 172.21.15.162:0/3340327985 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7804199e00 con 0x7f780406bdf0 2024-09-06T14:24:31.039 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:31.036+0000 7f7808f1b640 1 --2- 172.21.15.162:0/3340327985 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f780406bdf0 0x7f78041982d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f77fc005ab0 tx=0x7f77fc0329c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:24:31.039 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:31.036+0000 7f7801ffb640 1 -- 172.21.15.162:0/3340327985 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77fc03f030 con 0x7f780406bdf0 2024-09-06T14:24:31.039 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:31.037+0000 7f7801ffb640 1 -- 172.21.15.162:0/3340327985 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7f77fc0044c0 con 0x7f780406bdf0 2024-09-06T14:24:31.039 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:31.037+0000 7f7801ffb640 1 -- 172.21.15.162:0/3340327985 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77fc035070 con 0x7f780406bdf0 2024-09-06T14:24:31.039 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:31.037+0000 7f780b17f640 1 -- 172.21.15.162:0/3340327985 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f780419a090 con 0x7f780406bdf0 2024-09-06T14:24:31.039 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:31.037+0000 7f780b17f640 1 -- 172.21.15.162:0/3340327985 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78041a07d0 con 0x7f780406bdf0 2024-09-06T14:24:31.040 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:31.038+0000 7f780b17f640 1 -- 172.21.15.162:0/3340327985 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7804105d40 con 0x7f780406bdf0 2024-09-06T14:24:31.040 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:31.038+0000 7f7801ffb640 1 -- 172.21.15.162:0/3340327985 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 15) ==== 99786+0+0 (secure 0 0 0) 0x7f77fc042030 con 0x7f780406bdf0 2024-09-06T14:24:31.040 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:31.039+0000 7f7801ffb640 1 --2- 172.21.15.162:0/3340327985 >> v2:172.21.15.142:6800/853729507 conn(0x7f77f0077530 0x7f77f00799f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:31.040 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:31.039+0000 7f7801ffb640 1 -- 172.21.15.162:0/3340327985 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5829+0+0 (secure 0 0 0) 0x7f77fc0b9be0 con 0x7f780406bdf0 2024-09-06T14:24:31.040 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:31.039+0000 7f7803fff640 1 --2- 172.21.15.162:0/3340327985 >> v2:172.21.15.142:6800/853729507 conn(0x7f77f0077530 0x7f77f00799f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:31.040 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:31.039+0000 7f7803fff640 1 --2- 172.21.15.162:0/3340327985 >> v2:172.21.15.142:6800/853729507 conn(0x7f77f0077530 0x7f77f00799f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f780406c2a0 tx=0x7f77f4009500 comp rx=0 tx=0).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:24:31.040 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:31.043+0000 7f7801ffb640 1 -- 172.21.15.162:0/3340327985 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7f77fc082800 con 0x7f780406bdf0 2024-09-06T14:24:31.205 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:31.209+0000 7f780b17f640 1 -- 172.21.15.162:0/3340327985 --> v2:172.21.15.142:6800/853729507 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi162=a", "target": ["mon-mgr", ""]}) -- 0x7f780402e4f0 con 0x7f77f0077530 2024-09-06T14:24:31.212 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:31.216+0000 7f7801ffb640 1 -- 172.21.15.162:0/3340327985 <== mgr.14146 v2:172.21.15.142:6800/853729507 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+31 (secure 0 0 0) 0x7f780402e4f0 con 0x7f77f0077530 2024-09-06T14:24:31.212 INFO:teuthology.orchestra.run.smithi162.stdout:Scheduled prometheus update... 2024-09-06T14:24:31.214 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:31.218+0000 7f780b17f640 1 -- 172.21.15.162:0/3340327985 >> v2:172.21.15.142:6800/853729507 conn(0x7f77f0077530 msgr2=0x7f77f00799f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:31.214 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:31.218+0000 7f780b17f640 1 --2- 172.21.15.162:0/3340327985 >> v2:172.21.15.142:6800/853729507 conn(0x7f77f0077530 0x7f77f00799f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f780406c2a0 tx=0x7f77f4009500 comp rx=0 tx=0).stop 2024-09-06T14:24:31.214 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:31.218+0000 7f780b17f640 1 -- 172.21.15.162:0/3340327985 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f780406bdf0 msgr2=0x7f78041982d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:31.214 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:31.218+0000 7f780b17f640 1 --2- 172.21.15.162:0/3340327985 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f780406bdf0 0x7f78041982d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f77fc005ab0 tx=0x7f77fc0329c0 comp rx=0 tx=0).stop 2024-09-06T14:24:31.214 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:31.218+0000 7f780b17f640 1 -- 172.21.15.162:0/3340327985 shutdown_connections 2024-09-06T14:24:31.214 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:31.218+0000 7f780b17f640 1 --2- 172.21.15.162:0/3340327985 >> v2:172.21.15.142:6800/853729507 conn(0x7f77f0077530 0x7f77f00799f0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:31.214 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:31.218+0000 7f780b17f640 1 --2- 172.21.15.162:0/3340327985 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7804108490 0x7f7804199560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:31.215 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:31.218+0000 7f780b17f640 1 --2- 172.21.15.162:0/3340327985 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f78041058e0 0x7f7804198920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:31.215 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:31.218+0000 7f780b17f640 1 --2- 172.21.15.162:0/3340327985 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f780406bdf0 0x7f78041982d0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:31.215 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:31.218+0000 7f780b17f640 1 -- 172.21.15.162:0/3340327985 >> 172.21.15.162:0/3340327985 conn(0x7f78040fcf60 msgr2=0x7f7804103b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:24:31.215 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:31.218+0000 7f780b17f640 1 -- 172.21.15.162:0/3340327985 shutdown_connections 2024-09-06T14:24:31.215 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:31.218+0000 7f780b17f640 1 -- 172.21.15.162:0/3340327985 wait complete. 2024-09-06T14:24:31.458 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:31 smithi142 ceph-mon[37372]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-06T14:24:31.459 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:31 smithi142 ceph-mon[37372]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-06T14:24:31.459 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:31 smithi142 ceph-mon[37372]: osdmap e57: 8 total, 8 up, 8 in 2024-09-06T14:24:31.459 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:31 smithi142 ceph-mon[37372]: pgmap v183: 100 pgs: 12 creating+peering, 23 unknown, 65 active+clean; 578 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-09-06T14:24:31.459 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:31 smithi142 ceph-mon[29958]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-06T14:24:31.459 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:31 smithi142 ceph-mon[29958]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-06T14:24:31.459 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:31 smithi142 ceph-mon[29958]: osdmap e57: 8 total, 8 up, 8 in 2024-09-06T14:24:31.459 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:31 smithi142 ceph-mon[29958]: pgmap v183: 100 pgs: 12 creating+peering, 23 unknown, 65 active+clean; 578 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-09-06T14:24:31.919 DEBUG:teuthology.orchestra.run.smithi162:prometheus.a> sudo journalctl -f -n 0 -u ceph-ce32462e-6c5a-11ef-bce4-c7b262605968@prometheus.a.service 2024-09-06T14:24:31.921 INFO:tasks.cephadm:Adding node-exporter.a on smithi142 2024-09-06T14:24:31.921 INFO:tasks.cephadm:Adding node-exporter.b on smithi162 2024-09-06T14:24:31.921 DEBUG:teuthology.orchestra.run.smithi162:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph orch apply node-exporter '2;smithi142=a;smithi162=b' 2024-09-06T14:24:32.212 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.b/config 2024-09-06T14:24:32.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:32 smithi162 ceph-mon[33787]: osdmap e58: 8 total, 8 up, 8 in 2024-09-06T14:24:32.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:32 smithi162 ceph-mon[33787]: from='client.? 172.21.15.142:0/3263656736' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-06T14:24:32.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:32 smithi162 ceph-mon[33787]: from='client.? 172.21.15.142:0/4111201236' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-06T14:24:32.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:32 smithi162 ceph-mon[33787]: from='client.? ' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-06T14:24:32.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:32 smithi162 ceph-mon[33787]: from='client.? ' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-06T14:24:32.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:32 smithi162 ceph-mon[33787]: from='client.24373 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi162=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-06T14:24:32.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:32 smithi162 ceph-mon[33787]: Saving service prometheus spec with placement smithi162=a;count:1 2024-09-06T14:24:32.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:32 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:32.458 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:32 smithi142 ceph-mon[29958]: osdmap e58: 8 total, 8 up, 8 in 2024-09-06T14:24:32.459 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:32 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/3263656736' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-06T14:24:32.459 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:32 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/4111201236' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-06T14:24:32.459 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:32 smithi142 ceph-mon[29958]: from='client.? ' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-06T14:24:32.459 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:32 smithi142 ceph-mon[29958]: from='client.? ' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-06T14:24:32.459 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:32 smithi142 ceph-mon[29958]: from='client.24373 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi162=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-06T14:24:32.459 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:32 smithi142 ceph-mon[29958]: Saving service prometheus spec with placement smithi162=a;count:1 2024-09-06T14:24:32.459 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:32 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:32.459 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:32 smithi142 ceph-mon[37372]: osdmap e58: 8 total, 8 up, 8 in 2024-09-06T14:24:32.459 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:32 smithi142 ceph-mon[37372]: from='client.? 172.21.15.142:0/3263656736' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-06T14:24:32.459 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:32 smithi142 ceph-mon[37372]: from='client.? 172.21.15.142:0/4111201236' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-06T14:24:32.459 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:32 smithi142 ceph-mon[37372]: from='client.? ' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-06T14:24:32.460 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:32 smithi142 ceph-mon[37372]: from='client.? ' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-06T14:24:32.460 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:32 smithi142 ceph-mon[37372]: from='client.24373 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi162=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-06T14:24:32.460 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:32 smithi142 ceph-mon[37372]: Saving service prometheus spec with placement smithi162=a;count:1 2024-09-06T14:24:32.460 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:32 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:32.695 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:32.695+0000 7f31d3f75640 1 -- 172.21.15.162:0/1466136443 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f31cc100b70 msgr2=0x7f31cc100f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:32.696 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:32.695+0000 7f31d3f75640 1 --2- 172.21.15.162:0/1466136443 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f31cc100b70 0x7f31cc100f50 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f31c8009900 tx=0x7f31c802a3f0 comp rx=0 tx=0).stop 2024-09-06T14:24:32.696 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:32.695+0000 7f31d3f75640 1 -- 172.21.15.162:0/1466136443 shutdown_connections 2024-09-06T14:24:32.696 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:32.695+0000 7f31d3f75640 1 --2- 172.21.15.162:0/1466136443 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f31cc108570 0x7f31cc10a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:32.696 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:32.695+0000 7f31d3f75640 1 --2- 172.21.15.162:0/1466136443 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f31cc105c40 0x7f31cc108030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:32.696 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:32.695+0000 7f31d3f75640 1 --2- 172.21.15.162:0/1466136443 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f31cc100b70 0x7f31cc100f50 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:32.696 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:32.695+0000 7f31d3f75640 1 -- 172.21.15.162:0/1466136443 >> 172.21.15.162:0/1466136443 conn(0x7f31cc0fc6a0 msgr2=0x7f31cc0feb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:24:32.696 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:32.695+0000 7f31d3f75640 1 -- 172.21.15.162:0/1466136443 shutdown_connections 2024-09-06T14:24:32.697 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:32.695+0000 7f31d3f75640 1 -- 172.21.15.162:0/1466136443 wait complete. 2024-09-06T14:24:32.697 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:32.696+0000 7f31d3f75640 1 Processor -- start 2024-09-06T14:24:32.697 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:32.696+0000 7f31d3f75640 1 -- start start 2024-09-06T14:24:32.697 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:32.696+0000 7f31d3f75640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f31cc100b70 0x7f31cc198160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:32.697 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:32.696+0000 7f31d3f75640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f31cc105c40 0x7f31cc1987b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:32.697 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:32.696+0000 7f31d3f75640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f31cc108570 0x7f31cc199380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:32.698 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:32.696+0000 7f31d3f75640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f31cc10da60 con 0x7f31cc105c40 2024-09-06T14:24:32.698 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:32.696+0000 7f31d3f75640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f31cc10d8e0 con 0x7f31cc108570 2024-09-06T14:24:32.698 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:32.696+0000 7f31d3f75640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7f31cc10dbe0 con 0x7f31cc100b70 2024-09-06T14:24:32.698 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:32.696+0000 7f31d2512640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f31cc108570 0x7f31cc199380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:32.698 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:32.696+0000 7f31d2512640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f31cc108570 0x7f31cc199380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:39516/0 (socket says 172.21.15.162:39516) 2024-09-06T14:24:32.698 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:32.696+0000 7f31d2512640 1 -- 172.21.15.162:0/1324714711 learned_addr learned my addr 172.21.15.162:0/1324714711 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-09-06T14:24:32.698 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:32.696+0000 7f31d1d11640 1 --2- 172.21.15.162:0/1324714711 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f31cc100b70 0x7f31cc198160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:32.699 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:32.696+0000 7f31d1510640 1 --2- 172.21.15.162:0/1324714711 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f31cc105c40 0x7f31cc1987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:32.699 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:32.697+0000 7f31d2512640 1 -- 172.21.15.162:0/1324714711 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f31cc100b70 msgr2=0x7f31cc198160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:32.699 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:32.697+0000 7f31d2512640 1 --2- 172.21.15.162:0/1324714711 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f31cc100b70 0x7f31cc198160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:32.699 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:32.697+0000 7f31d2512640 1 -- 172.21.15.162:0/1324714711 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f31cc105c40 msgr2=0x7f31cc1987b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:32.699 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:32.697+0000 7f31d2512640 1 --2- 172.21.15.162:0/1324714711 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f31cc105c40 0x7f31cc1987b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:32.699 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:32.697+0000 7f31d2512640 1 -- 172.21.15.162:0/1324714711 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31cc199c20 con 0x7f31cc108570 2024-09-06T14:24:32.699 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:32.697+0000 7f31d1510640 1 --2- 172.21.15.162:0/1324714711 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f31cc105c40 0x7f31cc1987b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-06T14:24:32.700 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:32.697+0000 7f31d2512640 1 --2- 172.21.15.162:0/1324714711 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f31cc108570 0x7f31cc199380 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f31c400cce0 tx=0x7f31c400a620 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:24:32.700 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:32.697+0000 7f31c2ffd640 1 -- 172.21.15.162:0/1324714711 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31c400f450 con 0x7f31cc108570 2024-09-06T14:24:32.700 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:32.697+0000 7f31d3f75640 1 -- 172.21.15.162:0/1324714711 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31cc199f10 con 0x7f31cc108570 2024-09-06T14:24:32.700 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:32.697+0000 7f31c2ffd640 1 -- 172.21.15.162:0/1324714711 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7f31c400fa70 con 0x7f31cc108570 2024-09-06T14:24:32.700 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:32.697+0000 7f31c2ffd640 1 -- 172.21.15.162:0/1324714711 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31c40189d0 con 0x7f31cc108570 2024-09-06T14:24:32.700 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:32.697+0000 7f31d3f75640 1 -- 172.21.15.162:0/1324714711 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31cc1a05f0 con 0x7f31cc108570 2024-09-06T14:24:32.700 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:32.698+0000 7f31d3f75640 1 -- 172.21.15.162:0/1324714711 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31cc100f50 con 0x7f31cc108570 2024-09-06T14:24:32.701 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:32.699+0000 7f31c2ffd640 1 -- 172.21.15.162:0/1324714711 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 15) ==== 99786+0+0 (secure 0 0 0) 0x7f31c400f5f0 con 0x7f31cc108570 2024-09-06T14:24:32.701 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:32.699+0000 7f31c2ffd640 1 --2- 172.21.15.162:0/1324714711 >> v2:172.21.15.142:6800/853729507 conn(0x7f31b8077530 0x7f31b80799f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:32.701 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:32.699+0000 7f31c2ffd640 1 -- 172.21.15.162:0/1324714711 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 5840+0+0 (secure 0 0 0) 0x7f31c4099af0 con 0x7f31cc108570 2024-09-06T14:24:32.701 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:32.699+0000 7f31d1d11640 1 --2- 172.21.15.162:0/1324714711 >> v2:172.21.15.142:6800/853729507 conn(0x7f31b8077530 0x7f31b80799f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:32.702 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:32.700+0000 7f31d1d11640 1 --2- 172.21.15.162:0/1324714711 >> v2:172.21.15.142:6800/853729507 conn(0x7f31b8077530 0x7f31b80799f0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f31c8000c00 tx=0x7f31c80058d0 comp rx=0 tx=0).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:24:32.702 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:32.703+0000 7f31c2ffd640 1 -- 172.21.15.162:0/1324714711 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7f31c4014030 con 0x7f31cc108570 2024-09-06T14:24:32.862 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:32.866+0000 7f31d3f75640 1 -- 172.21.15.162:0/1324714711 --> v2:172.21.15.142:6800/853729507 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi142=a;smithi162=b", "target": ["mon-mgr", ""]}) -- 0x7f31cc10a070 con 0x7f31b8077530 2024-09-06T14:24:32.870 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:32.874+0000 7f31c2ffd640 1 -- 172.21.15.162:0/1324714711 <== mgr.14146 v2:172.21.15.142:6800/853729507 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+34 (secure 0 0 0) 0x7f31cc10a070 con 0x7f31b8077530 2024-09-06T14:24:32.871 INFO:teuthology.orchestra.run.smithi162.stdout:Scheduled node-exporter update... 2024-09-06T14:24:32.873 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:32.876+0000 7f31d3f75640 1 -- 172.21.15.162:0/1324714711 >> v2:172.21.15.142:6800/853729507 conn(0x7f31b8077530 msgr2=0x7f31b80799f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:32.873 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:32.876+0000 7f31d3f75640 1 --2- 172.21.15.162:0/1324714711 >> v2:172.21.15.142:6800/853729507 conn(0x7f31b8077530 0x7f31b80799f0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f31c8000c00 tx=0x7f31c80058d0 comp rx=0 tx=0).stop 2024-09-06T14:24:32.873 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:32.876+0000 7f31d3f75640 1 -- 172.21.15.162:0/1324714711 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f31cc108570 msgr2=0x7f31cc199380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:32.873 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:32.876+0000 7f31d3f75640 1 --2- 172.21.15.162:0/1324714711 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f31cc108570 0x7f31cc199380 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f31c400cce0 tx=0x7f31c400a620 comp rx=0 tx=0).stop 2024-09-06T14:24:32.873 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:32.876+0000 7f31d3f75640 1 -- 172.21.15.162:0/1324714711 shutdown_connections 2024-09-06T14:24:32.873 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:32.876+0000 7f31d3f75640 1 --2- 172.21.15.162:0/1324714711 >> v2:172.21.15.142:6800/853729507 conn(0x7f31b8077530 0x7f31b80799f0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:32.873 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:32.876+0000 7f31d3f75640 1 --2- 172.21.15.162:0/1324714711 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f31cc108570 0x7f31cc199380 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:32.873 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:32.876+0000 7f31d3f75640 1 --2- 172.21.15.162:0/1324714711 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f31cc105c40 0x7f31cc1987b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:32.873 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:32.876+0000 7f31d3f75640 1 --2- 172.21.15.162:0/1324714711 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f31cc100b70 0x7f31cc198160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:32.873 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:32.876+0000 7f31d3f75640 1 -- 172.21.15.162:0/1324714711 >> 172.21.15.162:0/1324714711 conn(0x7f31cc0fc6a0 msgr2=0x7f31cc108950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:24:32.874 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:32.876+0000 7f31d3f75640 1 -- 172.21.15.162:0/1324714711 shutdown_connections 2024-09-06T14:24:32.874 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:32.876+0000 7f31d3f75640 1 -- 172.21.15.162:0/1324714711 wait complete. 2024-09-06T14:24:33.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:33 smithi162 ceph-mon[33787]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-06T14:24:33.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:33 smithi162 ceph-mon[33787]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-06T14:24:33.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:33 smithi162 ceph-mon[33787]: osdmap e59: 8 total, 8 up, 8 in 2024-09-06T14:24:33.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:33 smithi162 ceph-mon[33787]: from='client.? 172.21.15.142:0/4111201236' entity='client.rgw.foo.a' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-06T14:24:33.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:33 smithi162 ceph-mon[33787]: from='client.? 172.21.15.142:0/3263656736' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-06T14:24:33.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:33 smithi162 ceph-mon[33787]: from='client.? ' entity='client.rgw.foo.a' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-06T14:24:33.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:33 smithi162 ceph-mon[33787]: from='client.? ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-06T14:24:33.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:33 smithi162 ceph-mon[33787]: from='client.24379 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi142=a;smithi162=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-06T14:24:33.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:33 smithi162 ceph-mon[33787]: Saving service node-exporter spec with placement smithi142=a;smithi162=b;count:2 2024-09-06T14:24:33.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:33 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:33.298 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:33 smithi162 ceph-mon[33787]: pgmap v186: 132 pgs: 13 creating+peering, 20 unknown, 99 active+clean; 578 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-09-06T14:24:33.458 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:33 smithi142 ceph-mon[29958]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-06T14:24:33.458 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:33 smithi142 ceph-mon[29958]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-06T14:24:33.459 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:33 smithi142 ceph-mon[29958]: osdmap e59: 8 total, 8 up, 8 in 2024-09-06T14:24:33.459 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:33 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/4111201236' entity='client.rgw.foo.a' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-06T14:24:33.459 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:33 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/3263656736' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-06T14:24:33.459 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:33 smithi142 ceph-mon[29958]: from='client.? ' entity='client.rgw.foo.a' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-06T14:24:33.459 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:33 smithi142 ceph-mon[29958]: from='client.? ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-06T14:24:33.459 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:33 smithi142 ceph-mon[29958]: from='client.24379 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi142=a;smithi162=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-06T14:24:33.459 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:33 smithi142 ceph-mon[29958]: Saving service node-exporter spec with placement smithi142=a;smithi162=b;count:2 2024-09-06T14:24:33.459 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:33 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:33.459 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:33 smithi142 ceph-mon[29958]: pgmap v186: 132 pgs: 13 creating+peering, 20 unknown, 99 active+clean; 578 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-09-06T14:24:33.459 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:33 smithi142 ceph-mon[37372]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-06T14:24:33.459 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:33 smithi142 ceph-mon[37372]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-06T14:24:33.459 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:33 smithi142 ceph-mon[37372]: osdmap e59: 8 total, 8 up, 8 in 2024-09-06T14:24:33.460 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:33 smithi142 ceph-mon[37372]: from='client.? 172.21.15.142:0/4111201236' entity='client.rgw.foo.a' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-06T14:24:33.460 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:33 smithi142 ceph-mon[37372]: from='client.? 172.21.15.142:0/3263656736' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-06T14:24:33.460 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:33 smithi142 ceph-mon[37372]: from='client.? ' entity='client.rgw.foo.a' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-06T14:24:33.460 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:33 smithi142 ceph-mon[37372]: from='client.? ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-06T14:24:33.460 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:33 smithi142 ceph-mon[37372]: from='client.24379 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi142=a;smithi162=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-06T14:24:33.460 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:33 smithi142 ceph-mon[37372]: Saving service node-exporter spec with placement smithi142=a;smithi162=b;count:2 2024-09-06T14:24:33.460 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:33 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:33.460 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:33 smithi142 ceph-mon[37372]: pgmap v186: 132 pgs: 13 creating+peering, 20 unknown, 99 active+clean; 578 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-09-06T14:24:33.724 DEBUG:teuthology.orchestra.run.smithi142:node-exporter.a> sudo journalctl -f -n 0 -u ceph-ce32462e-6c5a-11ef-bce4-c7b262605968@node-exporter.a.service 2024-09-06T14:24:33.727 DEBUG:teuthology.orchestra.run.smithi162:node-exporter.b> sudo journalctl -f -n 0 -u ceph-ce32462e-6c5a-11ef-bce4-c7b262605968@node-exporter.b.service 2024-09-06T14:24:33.729 INFO:tasks.cephadm:Adding alertmanager.a on smithi142 2024-09-06T14:24:33.729 DEBUG:teuthology.orchestra.run.smithi162:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph orch apply alertmanager '1;smithi142=a' 2024-09-06T14:24:34.285 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:34 smithi162 ceph-mon[33787]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-06T14:24:34.285 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:34 smithi162 ceph-mon[33787]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-06T14:24:34.285 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:34 smithi162 ceph-mon[33787]: osdmap e60: 8 total, 8 up, 8 in 2024-09-06T14:24:34.285 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:34 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:34.286 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:34 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:34.286 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:34 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:34.286 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:34 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]} : dispatch 2024-09-06T14:24:34.286 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:34 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-09-06T14:24:34.286 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:34 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:24:34.286 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:34 smithi162 ceph-mon[33787]: Deploying daemon iscsi.iscsi.a on smithi162 2024-09-06T14:24:34.286 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:34 smithi162 ceph-mon[33787]: osdmap e61: 8 total, 8 up, 8 in 2024-09-06T14:24:34.458 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:34 smithi142 ceph-mon[37372]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-06T14:24:34.459 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:34 smithi142 ceph-mon[37372]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-06T14:24:34.459 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:34 smithi142 ceph-mon[37372]: osdmap e60: 8 total, 8 up, 8 in 2024-09-06T14:24:34.459 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:34 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:34.459 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:34 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:34.459 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:34 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:34.459 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:34 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]} : dispatch 2024-09-06T14:24:34.459 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:34 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-09-06T14:24:34.459 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:34 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:24:34.459 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:34 smithi142 ceph-mon[37372]: Deploying daemon iscsi.iscsi.a on smithi162 2024-09-06T14:24:34.459 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:34 smithi142 ceph-mon[37372]: osdmap e61: 8 total, 8 up, 8 in 2024-09-06T14:24:34.460 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:34 smithi142 ceph-mon[29958]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-06T14:24:34.460 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:34 smithi142 ceph-mon[29958]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-06T14:24:34.460 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:34 smithi142 ceph-mon[29958]: osdmap e60: 8 total, 8 up, 8 in 2024-09-06T14:24:34.460 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:34 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:34.460 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:34 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:34.460 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:34 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:34.460 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:34 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]} : dispatch 2024-09-06T14:24:34.460 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:34 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-09-06T14:24:34.460 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:34 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:24:34.460 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:34 smithi142 ceph-mon[29958]: Deploying daemon iscsi.iscsi.a on smithi162 2024-09-06T14:24:34.460 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:34 smithi142 ceph-mon[29958]: osdmap e61: 8 total, 8 up, 8 in 2024-09-06T14:24:34.495 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.b/config 2024-09-06T14:24:34.950 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:34.951+0000 7fafd8e4c640 1 -- 172.21.15.162:0/3975116900 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fafd406b390 msgr2=0x7fafd41053a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:34.950 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:34.951+0000 7fafd8e4c640 1 --2- 172.21.15.162:0/3975116900 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fafd406b390 0x7fafd41053a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fafc4009ca0 tx=0x7fafc402a8c0 comp rx=0 tx=0).stop 2024-09-06T14:24:34.950 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:34.951+0000 7fafd8e4c640 1 -- 172.21.15.162:0/3975116900 shutdown_connections 2024-09-06T14:24:34.950 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:34.951+0000 7fafd8e4c640 1 --2- 172.21.15.162:0/3975116900 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fafd4108450 0x7fafd410a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:34.950 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:34.951+0000 7fafd8e4c640 1 --2- 172.21.15.162:0/3975116900 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fafd41058e0 0x7fafd4107d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:34.950 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:34.951+0000 7fafd8e4c640 1 --2- 172.21.15.162:0/3975116900 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fafd406b390 0x7fafd41053a0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:34.950 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:34.951+0000 7fafd8e4c640 1 -- 172.21.15.162:0/3975116900 >> 172.21.15.162:0/3975116900 conn(0x7fafd40fcf10 msgr2=0x7fafd40ff350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:24:34.950 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:34.951+0000 7fafd8e4c640 1 -- 172.21.15.162:0/3975116900 shutdown_connections 2024-09-06T14:24:34.950 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:34.951+0000 7fafd8e4c640 1 -- 172.21.15.162:0/3975116900 wait complete. 2024-09-06T14:24:34.950 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:34.952+0000 7fafd8e4c640 1 Processor -- start 2024-09-06T14:24:34.951 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:34.952+0000 7fafd8e4c640 1 -- start start 2024-09-06T14:24:34.951 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:34.952+0000 7fafd8e4c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fafd406b390 0x7fafd4198190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:34.951 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:34.952+0000 7fafd8e4c640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fafd41058e0 0x7fafd41987e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:34.951 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:34.952+0000 7fafd8e4c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fafd4108450 0x7fafd4199420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:34.951 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:34.952+0000 7fafd8e4c640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fafd410da50 con 0x7fafd406b390 2024-09-06T14:24:34.951 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:34.952+0000 7fafd8e4c640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fafd410d8d0 con 0x7fafd4108450 2024-09-06T14:24:34.951 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:34.952+0000 7fafd8e4c640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7fafd410dbd0 con 0x7fafd41058e0 2024-09-06T14:24:34.951 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:34.952+0000 7fafd2d9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fafd4108450 0x7fafd4199420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:34.951 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:34.952+0000 7fafd2d9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fafd4108450 0x7fafd4199420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:39540/0 (socket says 172.21.15.162:39540) 2024-09-06T14:24:34.951 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:34.952+0000 7fafd2d9d640 1 -- 172.21.15.162:0/3550524590 learned_addr learned my addr 172.21.15.162:0/3550524590 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-09-06T14:24:34.951 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:34.952+0000 7fafd259c640 1 --2- 172.21.15.162:0/3550524590 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fafd406b390 0x7fafd4198190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:34.951 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:34.952+0000 7fafd1d9b640 1 --2- 172.21.15.162:0/3550524590 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fafd41058e0 0x7fafd41987e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:34.952 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:34.952+0000 7fafd2d9d640 1 -- 172.21.15.162:0/3550524590 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fafd41058e0 msgr2=0x7fafd41987e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:34.952 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:34.952+0000 7fafd2d9d640 1 --2- 172.21.15.162:0/3550524590 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fafd41058e0 0x7fafd41987e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:34.952 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:34.952+0000 7fafd2d9d640 1 -- 172.21.15.162:0/3550524590 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fafd406b390 msgr2=0x7fafd4198190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:34.952 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:34.952+0000 7fafd2d9d640 1 --2- 172.21.15.162:0/3550524590 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fafd406b390 0x7fafd4198190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:34.952 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:34.952+0000 7fafd2d9d640 1 -- 172.21.15.162:0/3550524590 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafd4199cc0 con 0x7fafd4108450 2024-09-06T14:24:34.952 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:34.952+0000 7fafd2d9d640 1 --2- 172.21.15.162:0/3550524590 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fafd4108450 0x7fafd4199420 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fafcc00d750 tx=0x7fafcc00db50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:24:34.952 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:34.953+0000 7fafc37fe640 1 -- 172.21.15.162:0/3550524590 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafcc017070 con 0x7fafd4108450 2024-09-06T14:24:34.952 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:34.953+0000 7fafd8e4c640 1 -- 172.21.15.162:0/3550524590 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafd4199fb0 con 0x7fafd4108450 2024-09-06T14:24:34.952 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:34.953+0000 7fafc37fe640 1 -- 172.21.15.162:0/3550524590 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7fafcc0097c0 con 0x7fafd4108450 2024-09-06T14:24:34.952 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:34.953+0000 7fafc37fe640 1 -- 172.21.15.162:0/3550524590 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafcc004a80 con 0x7fafd4108450 2024-09-06T14:24:34.952 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:34.953+0000 7fafd8e4c640 1 -- 172.21.15.162:0/3550524590 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafd41a0690 con 0x7fafd4108450 2024-09-06T14:24:34.953 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:34.954+0000 7fafd8e4c640 1 -- 172.21.15.162:0/3550524590 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fafd4105d00 con 0x7fafd4108450 2024-09-06T14:24:34.953 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:34.954+0000 7fafc37fe640 1 -- 172.21.15.162:0/3550524590 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 15) ==== 99786+0+0 (secure 0 0 0) 0x7fafcc004c20 con 0x7fafd4108450 2024-09-06T14:24:34.953 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:34.955+0000 7fafc37fe640 1 --2- 172.21.15.162:0/3550524590 >> v2:172.21.15.142:6800/853729507 conn(0x7fafbc077530 0x7fafbc0799f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:34.953 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:34.955+0000 7fafc37fe640 1 -- 172.21.15.162:0/3550524590 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 5856+0+0 (secure 0 0 0) 0x7fafcc098ff0 con 0x7fafd4108450 2024-09-06T14:24:34.953 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:34.955+0000 7fafd259c640 1 --2- 172.21.15.162:0/3550524590 >> v2:172.21.15.142:6800/853729507 conn(0x7fafbc077530 0x7fafbc0799f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:34.953 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:34.956+0000 7fafd259c640 1 --2- 172.21.15.162:0/3550524590 >> v2:172.21.15.142:6800/853729507 conn(0x7fafbc077530 0x7fafbc0799f0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fafc4000f80 tx=0x7fafc4005ef0 comp rx=0 tx=0).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:24:34.955 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:34.959+0000 7fafc37fe640 1 -- 172.21.15.162:0/3550524590 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7fafcc061b80 con 0x7fafd4108450 2024-09-06T14:24:35.122 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:35.126+0000 7fafd8e4c640 1 -- 172.21.15.162:0/3550524590 --> v2:172.21.15.142:6800/853729507 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi142=a", "target": ["mon-mgr", ""]}) -- 0x7fafd402e650 con 0x7fafbc077530 2024-09-06T14:24:35.122 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:35 smithi162 ceph-mon[33787]: pgmap v189: 132 pgs: 8 creating+peering, 12 unknown, 112 active+clean; 578 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 255 B/s wr, 1 op/s 2024-09-06T14:24:35.129 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:35.133+0000 7fafc37fe640 1 -- 172.21.15.162:0/3550524590 <== mgr.14146 v2:172.21.15.142:6800/853729507 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+33 (secure 0 0 0) 0x7fafd402e650 con 0x7fafbc077530 2024-09-06T14:24:35.129 INFO:teuthology.orchestra.run.smithi162.stdout:Scheduled alertmanager update... 2024-09-06T14:24:35.131 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:35.135+0000 7fafd8e4c640 1 -- 172.21.15.162:0/3550524590 >> v2:172.21.15.142:6800/853729507 conn(0x7fafbc077530 msgr2=0x7fafbc0799f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:35.131 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:35.135+0000 7fafd8e4c640 1 --2- 172.21.15.162:0/3550524590 >> v2:172.21.15.142:6800/853729507 conn(0x7fafbc077530 0x7fafbc0799f0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fafc4000f80 tx=0x7fafc4005ef0 comp rx=0 tx=0).stop 2024-09-06T14:24:35.131 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:35.135+0000 7fafd8e4c640 1 -- 172.21.15.162:0/3550524590 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fafd4108450 msgr2=0x7fafd4199420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:35.131 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:35.135+0000 7fafd8e4c640 1 --2- 172.21.15.162:0/3550524590 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fafd4108450 0x7fafd4199420 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fafcc00d750 tx=0x7fafcc00db50 comp rx=0 tx=0).stop 2024-09-06T14:24:35.131 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:35.135+0000 7fafd8e4c640 1 -- 172.21.15.162:0/3550524590 shutdown_connections 2024-09-06T14:24:35.131 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:35.135+0000 7fafd8e4c640 1 --2- 172.21.15.162:0/3550524590 >> v2:172.21.15.142:6800/853729507 conn(0x7fafbc077530 0x7fafbc0799f0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:35.131 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:35.135+0000 7fafd8e4c640 1 --2- 172.21.15.162:0/3550524590 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fafd4108450 0x7fafd4199420 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:35.132 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:35.135+0000 7fafd8e4c640 1 --2- 172.21.15.162:0/3550524590 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fafd41058e0 0x7fafd41987e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:35.132 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:35.135+0000 7fafd8e4c640 1 --2- 172.21.15.162:0/3550524590 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fafd406b390 0x7fafd4198190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:35.132 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:35.135+0000 7fafd8e4c640 1 -- 172.21.15.162:0/3550524590 >> 172.21.15.162:0/3550524590 conn(0x7fafd40fcf10 msgr2=0x7fafd4103bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:24:35.132 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:35.135+0000 7fafd8e4c640 1 -- 172.21.15.162:0/3550524590 shutdown_connections 2024-09-06T14:24:35.132 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:35.135+0000 7fafd8e4c640 1 -- 172.21.15.162:0/3550524590 wait complete. 2024-09-06T14:24:35.458 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:35 smithi142 ceph-mon[29958]: pgmap v189: 132 pgs: 8 creating+peering, 12 unknown, 112 active+clean; 578 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 255 B/s wr, 1 op/s 2024-09-06T14:24:35.459 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:35 smithi142 ceph-mon[37372]: pgmap v189: 132 pgs: 8 creating+peering, 12 unknown, 112 active+clean; 578 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 255 B/s wr, 1 op/s 2024-09-06T14:24:36.242 DEBUG:teuthology.orchestra.run.smithi142:alertmanager.a> sudo journalctl -f -n 0 -u ceph-ce32462e-6c5a-11ef-bce4-c7b262605968@alertmanager.a.service 2024-09-06T14:24:36.245 INFO:tasks.cephadm:Adding grafana.a on smithi162 2024-09-06T14:24:36.245 DEBUG:teuthology.orchestra.run.smithi162:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph orch apply grafana '1;smithi162=a' 2024-09-06T14:24:36.269 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:36 smithi162 ceph-mon[33787]: from='client.24400 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi142=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-06T14:24:36.269 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:36 smithi162 ceph-mon[33787]: Saving service alertmanager spec with placement smithi142=a;count:1 2024-09-06T14:24:36.269 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:36 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:36.434 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:36 smithi142 ceph-mon[29958]: from='client.24400 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi142=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-06T14:24:36.434 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:36 smithi142 ceph-mon[29958]: Saving service alertmanager spec with placement smithi142=a;count:1 2024-09-06T14:24:36.434 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:36 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:36.435 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:36 smithi142 ceph-mon[37372]: from='client.24400 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi142=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-06T14:24:36.435 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:36 smithi142 ceph-mon[37372]: Saving service alertmanager spec with placement smithi142=a;count:1 2024-09-06T14:24:36.435 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:36 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:36.780 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 06 14:24:36 smithi162 systemd[1]: Started Ceph iscsi.iscsi.a for ce32462e-6c5a-11ef-bce4-c7b262605968. 2024-09-06T14:24:36.824 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.b/config 2024-09-06T14:24:37.047 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 06 14:24:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-iscsi-iscsi-a[59853]: debug Started the configuration object watcher 2024-09-06T14:24:37.047 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 06 14:24:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-iscsi-iscsi-a[59853]: debug Checking for config object changes every 1s 2024-09-06T14:24:37.047 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 06 14:24:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-iscsi-iscsi-a[59853]: debug Processing osd blocklist entries for this node 2024-09-06T14:24:37.355 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 06 14:24:37 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-iscsi-iscsi-a[59853]: debug Reading the configuration object to update local LIO configuration 2024-09-06T14:24:37.355 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 06 14:24:37 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-iscsi-iscsi-a[59853]: debug Configuration does not have an entry for this host(smithi162) - nothing to define to LIO 2024-09-06T14:24:37.355 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 06 14:24:37 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-iscsi-iscsi-a[59853]: * Serving Flask app 'rbd-target-api' (lazy loading) 2024-09-06T14:24:37.355 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 06 14:24:37 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-iscsi-iscsi-a[59853]: * Environment: production 2024-09-06T14:24:37.355 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 06 14:24:37 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-iscsi-iscsi-a[59853]: WARNING: This is a development server. Do not use it in a production deployment. 2024-09-06T14:24:37.355 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 06 14:24:37 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-iscsi-iscsi-a[59853]: Use a production WSGI server instead. 2024-09-06T14:24:37.355 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 06 14:24:37 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-iscsi-iscsi-a[59853]: * Debug mode: off 2024-09-06T14:24:37.355 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 06 14:24:37 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-iscsi-iscsi-a[59853]: debug * Running on all addresses. 2024-09-06T14:24:37.355 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 06 14:24:37 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-iscsi-iscsi-a[59853]: WARNING: This is a development server. Do not use it in a production deployment. 2024-09-06T14:24:37.355 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 06 14:24:37 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-iscsi-iscsi-a[59853]: * Running on all addresses. 2024-09-06T14:24:37.355 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 06 14:24:37 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-iscsi-iscsi-a[59853]: WARNING: This is a development server. Do not use it in a production deployment. 2024-09-06T14:24:37.355 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 06 14:24:37 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-iscsi-iscsi-a[59853]: debug * Running on http://[fe80::ec4:7aff:fe88:7f41]:5000/ (Press CTRL+C to quit) 2024-09-06T14:24:37.356 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 06 14:24:37 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-iscsi-iscsi-a[59853]: * Running on http://[fe80::ec4:7aff:fe88:7f41]:5000/ (Press CTRL+C to quit) 2024-09-06T14:24:37.542 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:37.545+0000 7f8436772640 1 -- 172.21.15.162:0/3805065861 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f843006bdf0 msgr2=0x7f8430105350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:37.542 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:37.545+0000 7f8436772640 1 --2- 172.21.15.162:0/3805065861 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f843006bdf0 0x7f8430105350 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f8420009900 tx=0x7f842002a410 comp rx=0 tx=0).stop 2024-09-06T14:24:37.542 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:37.546+0000 7f8436772640 1 -- 172.21.15.162:0/3805065861 shutdown_connections 2024-09-06T14:24:37.542 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:37.546+0000 7f8436772640 1 --2- 172.21.15.162:0/3805065861 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8430108490 0x7f843010a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:37.543 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:37.546+0000 7f8436772640 1 --2- 172.21.15.162:0/3805065861 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f84301058e0 0x7f8430107d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:37.543 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:37.546+0000 7f8436772640 1 --2- 172.21.15.162:0/3805065861 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f843006bdf0 0x7f8430105350 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:37.543 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:37.546+0000 7f8436772640 1 -- 172.21.15.162:0/3805065861 >> 172.21.15.162:0/3805065861 conn(0x7f84300fcf60 msgr2=0x7f84300ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:24:37.543 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:37.546+0000 7f8436772640 1 -- 172.21.15.162:0/3805065861 shutdown_connections 2024-09-06T14:24:37.543 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:37.546+0000 7f8436772640 1 -- 172.21.15.162:0/3805065861 wait complete. 2024-09-06T14:24:37.545 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:37.546+0000 7f8436772640 1 Processor -- start 2024-09-06T14:24:37.545 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:37.546+0000 7f8436772640 1 -- start start 2024-09-06T14:24:37.545 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:37.547+0000 7f8436772640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f843006bdf0 0x7f8430102a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:37.545 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:37.547+0000 7f8436772640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f84301058e0 0x7f84301012f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:37.545 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:37.547+0000 7f8436772640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8430108490 0x7f8430101830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:37.545 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:37.547+0000 7f8436772640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f843010db20 con 0x7f84301058e0 2024-09-06T14:24:37.545 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:37.547+0000 7f8436772640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f843010d9a0 con 0x7f843006bdf0 2024-09-06T14:24:37.545 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:37.547+0000 7f8436772640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7f843010dca0 con 0x7f8430108490 2024-09-06T14:24:37.545 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:37.547+0000 7f842ffff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f843006bdf0 0x7f8430102a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:37.546 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:37.547+0000 7f842ffff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f843006bdf0 0x7f8430102a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:39642/0 (socket says 172.21.15.162:39642) 2024-09-06T14:24:37.546 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:37.547+0000 7f842ffff640 1 -- 172.21.15.162:0/1729278696 learned_addr learned my addr 172.21.15.162:0/1729278696 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-09-06T14:24:37.546 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:37.547+0000 7f842f7fe640 1 --2- 172.21.15.162:0/1729278696 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f84301058e0 0x7f84301012f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:37.546 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:37.547+0000 7f8434d0f640 1 --2- 172.21.15.162:0/1729278696 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8430108490 0x7f8430101830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:37.546 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:37.547+0000 7f842ffff640 1 -- 172.21.15.162:0/1729278696 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8430108490 msgr2=0x7f8430101830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:37.546 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:37.547+0000 7f842ffff640 1 --2- 172.21.15.162:0/1729278696 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8430108490 0x7f8430101830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:37.546 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:37.547+0000 7f842ffff640 1 -- 172.21.15.162:0/1729278696 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f84301058e0 msgr2=0x7f84301012f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:37.546 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:37.547+0000 7f842ffff640 1 --2- 172.21.15.162:0/1729278696 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f84301058e0 0x7f84301012f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:37.546 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:37.547+0000 7f842ffff640 1 -- 172.21.15.162:0/1729278696 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84301020f0 con 0x7f843006bdf0 2024-09-06T14:24:37.546 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:37.547+0000 7f842f7fe640 1 --2- 172.21.15.162:0/1729278696 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f84301058e0 0x7f84301012f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-06T14:24:37.546 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:37.547+0000 7f842ffff640 1 --2- 172.21.15.162:0/1729278696 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f843006bdf0 0x7f8430102a80 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8420006010 tx=0x7f8420032c50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:24:37.546 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:37.548+0000 7f842d7fa640 1 -- 172.21.15.162:0/1729278696 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f842003f030 con 0x7f843006bdf0 2024-09-06T14:24:37.546 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:37.548+0000 7f842d7fa640 1 -- 172.21.15.162:0/1729278696 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7f8420005480 con 0x7f843006bdf0 2024-09-06T14:24:37.547 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:37.548+0000 7f8436772640 1 -- 172.21.15.162:0/1729278696 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84301a23c0 con 0x7f843006bdf0 2024-09-06T14:24:37.547 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:37.548+0000 7f842d7fa640 1 -- 172.21.15.162:0/1729278696 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8420035070 con 0x7f843006bdf0 2024-09-06T14:24:37.547 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:37.548+0000 7f8436772640 1 -- 172.21.15.162:0/1729278696 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84301a2870 con 0x7f843006bdf0 2024-09-06T14:24:37.547 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:37.549+0000 7f8436772640 1 -- 172.21.15.162:0/1729278696 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8430105d40 con 0x7f843006bdf0 2024-09-06T14:24:37.547 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:37.549+0000 7f842d7fa640 1 -- 172.21.15.162:0/1729278696 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 15) ==== 99786+0+0 (secure 0 0 0) 0x7f8420042030 con 0x7f843006bdf0 2024-09-06T14:24:37.548 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:37.550+0000 7f842d7fa640 1 --2- 172.21.15.162:0/1729278696 >> v2:172.21.15.142:6800/853729507 conn(0x7f8418077530 0x7f84180799f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:37.548 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:37.550+0000 7f842d7fa640 1 -- 172.21.15.162:0/1729278696 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 5856+0+0 (secure 0 0 0) 0x7f84200b9ec0 con 0x7f843006bdf0 2024-09-06T14:24:37.548 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:37.550+0000 7f842f7fe640 1 --2- 172.21.15.162:0/1729278696 >> v2:172.21.15.142:6800/853729507 conn(0x7f8418077530 0x7f84180799f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:37.548 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:37.551+0000 7f842f7fe640 1 --2- 172.21.15.162:0/1729278696 >> v2:172.21.15.142:6800/853729507 conn(0x7f8418077530 0x7f84180799f0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f84240058c0 tx=0x7f842400a2d0 comp rx=0 tx=0).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:24:37.550 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:37.554+0000 7f842d7fa640 1 -- 172.21.15.162:0/1729278696 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7f8420082a50 con 0x7f843006bdf0 2024-09-06T14:24:37.721 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:37.724+0000 7f8436772640 1 -- 172.21.15.162:0/1729278696 --> v2:172.21.15.142:6800/853729507 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi162=a", "target": ["mon-mgr", ""]}) -- 0x7f84300632f0 con 0x7f8418077530 2024-09-06T14:24:37.726 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:37.730+0000 7f842d7fa640 1 -- 172.21.15.162:0/1729278696 <== mgr.14146 v2:172.21.15.142:6800/853729507 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7f84300632f0 con 0x7f8418077530 2024-09-06T14:24:37.727 INFO:teuthology.orchestra.run.smithi162.stdout:Scheduled grafana update... 2024-09-06T14:24:37.729 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:37.733+0000 7f8436772640 1 -- 172.21.15.162:0/1729278696 >> v2:172.21.15.142:6800/853729507 conn(0x7f8418077530 msgr2=0x7f84180799f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:37.729 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:37.733+0000 7f8436772640 1 --2- 172.21.15.162:0/1729278696 >> v2:172.21.15.142:6800/853729507 conn(0x7f8418077530 0x7f84180799f0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f84240058c0 tx=0x7f842400a2d0 comp rx=0 tx=0).stop 2024-09-06T14:24:37.729 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:37.733+0000 7f8436772640 1 -- 172.21.15.162:0/1729278696 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f843006bdf0 msgr2=0x7f8430102a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:37.729 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:37.733+0000 7f8436772640 1 --2- 172.21.15.162:0/1729278696 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f843006bdf0 0x7f8430102a80 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8420006010 tx=0x7f8420032c50 comp rx=0 tx=0).stop 2024-09-06T14:24:37.729 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:37.733+0000 7f8436772640 1 -- 172.21.15.162:0/1729278696 shutdown_connections 2024-09-06T14:24:37.729 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:37.733+0000 7f8436772640 1 --2- 172.21.15.162:0/1729278696 >> v2:172.21.15.142:6800/853729507 conn(0x7f8418077530 0x7f84180799f0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:37.730 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:37.733+0000 7f8436772640 1 --2- 172.21.15.162:0/1729278696 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8430108490 0x7f8430101830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:37.730 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:37.733+0000 7f8436772640 1 --2- 172.21.15.162:0/1729278696 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f84301058e0 0x7f84301012f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:37.730 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:37.733+0000 7f8436772640 1 --2- 172.21.15.162:0/1729278696 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f843006bdf0 0x7f8430102a80 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:37.730 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:37.733+0000 7f8436772640 1 -- 172.21.15.162:0/1729278696 >> 172.21.15.162:0/1729278696 conn(0x7f84300fcf60 msgr2=0x7f8430103b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:24:37.730 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:37.733+0000 7f8436772640 1 -- 172.21.15.162:0/1729278696 shutdown_connections 2024-09-06T14:24:37.730 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:37.733+0000 7f8436772640 1 -- 172.21.15.162:0/1729278696 wait complete. 2024-09-06T14:24:38.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:37 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:38.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:37 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:38.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:37 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:38.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:37 smithi162 ceph-mon[33787]: Checking pool "datapool" exists for service iscsi.datapool 2024-09-06T14:24:38.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:37 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:38.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:37 smithi162 ceph-mon[33787]: pgmap v190: 132 pgs: 132 active+clean; 581 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s rd, 4.1 KiB/s wr, 124 op/s 2024-09-06T14:24:38.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:37 smithi162 ceph-mon[33787]: from='client.? 172.21.15.162:0/1351849023' entity='client.iscsi.iscsi.a' cmd={"prefix": "osd blocklist ls"} : dispatch 2024-09-06T14:24:38.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:37 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:38.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:37 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:38.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:37 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:38.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:37 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:38.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:37 smithi142 ceph-mon[29958]: Checking pool "datapool" exists for service iscsi.datapool 2024-09-06T14:24:38.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:37 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:38.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:37 smithi142 ceph-mon[29958]: pgmap v190: 132 pgs: 132 active+clean; 581 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s rd, 4.1 KiB/s wr, 124 op/s 2024-09-06T14:24:38.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:37 smithi142 ceph-mon[29958]: from='client.? 172.21.15.162:0/1351849023' entity='client.iscsi.iscsi.a' cmd={"prefix": "osd blocklist ls"} : dispatch 2024-09-06T14:24:38.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:37 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:38.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:37 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:38.210 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:37 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:38.210 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:37 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:38.210 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:37 smithi142 ceph-mon[37372]: Checking pool "datapool" exists for service iscsi.datapool 2024-09-06T14:24:38.210 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:37 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:38.210 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:37 smithi142 ceph-mon[37372]: pgmap v190: 132 pgs: 132 active+clean; 581 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s rd, 4.1 KiB/s wr, 124 op/s 2024-09-06T14:24:38.210 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:37 smithi142 ceph-mon[37372]: from='client.? 172.21.15.162:0/1351849023' entity='client.iscsi.iscsi.a' cmd={"prefix": "osd blocklist ls"} : dispatch 2024-09-06T14:24:38.210 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:37 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:38.486 DEBUG:teuthology.orchestra.run.smithi162:grafana.a> sudo journalctl -f -n 0 -u ceph-ce32462e-6c5a-11ef-bce4-c7b262605968@grafana.a.service 2024-09-06T14:24:38.529 INFO:tasks.cephadm:Setting up client nodes... 2024-09-06T14:24:38.529 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-06T14:24:38.786 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:24:39.178 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:38 smithi142 ceph-mon[29958]: Deploying daemon prometheus.a on smithi162 2024-09-06T14:24:39.179 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:38 smithi142 ceph-mon[29958]: from='client.24436 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi162=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-06T14:24:39.179 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:38 smithi142 ceph-mon[29958]: Saving service grafana spec with placement smithi162=a;count:1 2024-09-06T14:24:39.179 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:38 smithi142 ceph-mon[29958]: mgrmap e16: y(active, since 4m), standbys: x 2024-09-06T14:24:39.179 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:38 smithi142 ceph-mon[37372]: Deploying daemon prometheus.a on smithi162 2024-09-06T14:24:39.179 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:38 smithi142 ceph-mon[37372]: from='client.24436 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi162=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-06T14:24:39.179 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:38 smithi142 ceph-mon[37372]: Saving service grafana spec with placement smithi162=a;count:1 2024-09-06T14:24:39.179 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:38 smithi142 ceph-mon[37372]: mgrmap e16: y(active, since 4m), standbys: x 2024-09-06T14:24:39.271 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:39.272+0000 7fa234900640 1 -- 172.21.15.142:0/345945659 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa230102760 msgr2=0x7fa230102b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:39.271 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:39.272+0000 7fa234900640 1 --2- 172.21.15.142:0/345945659 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa230102760 0x7fa230102b60 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fa2200098c0 tx=0x7fa22002a390 comp rx=0 tx=0).stop 2024-09-06T14:24:39.271 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:39.272+0000 7fa234900640 1 -- 172.21.15.142:0/345945659 shutdown_connections 2024-09-06T14:24:39.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:39.272+0000 7fa234900640 1 --2- 172.21.15.142:0/345945659 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa230104280 0x7fa23010ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:39.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:39.272+0000 7fa234900640 1 --2- 172.21.15.142:0/345945659 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa2301038c0 0x7fa230103d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:39.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:39.272+0000 7fa234900640 1 --2- 172.21.15.142:0/345945659 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa230102760 0x7fa230102b60 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:39.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:39.272+0000 7fa234900640 1 -- 172.21.15.142:0/345945659 >> 172.21.15.142:0/345945659 conn(0x7fa2300fe050 msgr2=0x7fa230100470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:24:39.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:39.272+0000 7fa234900640 1 -- 172.21.15.142:0/345945659 shutdown_connections 2024-09-06T14:24:39.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:39.272+0000 7fa234900640 1 -- 172.21.15.142:0/345945659 wait complete. 2024-09-06T14:24:39.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:39.273+0000 7fa234900640 1 Processor -- start 2024-09-06T14:24:39.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:39.273+0000 7fa234900640 1 -- start start 2024-09-06T14:24:39.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:39.273+0000 7fa234900640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa230102760 0x7fa230076a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:39.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:39.273+0000 7fa234900640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa2301038c0 0x7fa2300752a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:39.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:39.273+0000 7fa234900640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa230104280 0x7fa2300757e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:39.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:39.273+0000 7fa234900640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fa23010daa0 con 0x7fa230102760 2024-09-06T14:24:39.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:39.273+0000 7fa234900640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa23010d920 con 0x7fa230104280 2024-09-06T14:24:39.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:39.273+0000 7fa234900640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7fa23010dc20 con 0x7fa2301038c0 2024-09-06T14:24:39.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:39.273+0000 7fa22e59c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa230102760 0x7fa230076a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:39.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:39.274+0000 7fa22e59c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa230102760 0x7fa230076a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:43368/0 (socket says 172.21.15.142:43368) 2024-09-06T14:24:39.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:39.274+0000 7fa22e59c640 1 -- 172.21.15.142:0/3429609051 learned_addr learned my addr 172.21.15.142:0/3429609051 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:24:39.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:39.274+0000 7fa22dd9b640 1 --2- 172.21.15.142:0/3429609051 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa2301038c0 0x7fa2300752a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:39.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:39.274+0000 7fa22ed9d640 1 --2- 172.21.15.142:0/3429609051 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa230104280 0x7fa2300757e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:39.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:39.274+0000 7fa22e59c640 1 -- 172.21.15.142:0/3429609051 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa2301038c0 msgr2=0x7fa2300752a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:39.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:39.274+0000 7fa22e59c640 1 --2- 172.21.15.142:0/3429609051 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa2301038c0 0x7fa2300752a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:39.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:39.274+0000 7fa22e59c640 1 -- 172.21.15.142:0/3429609051 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa230104280 msgr2=0x7fa2300757e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:39.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:39.274+0000 7fa22e59c640 1 --2- 172.21.15.142:0/3429609051 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa230104280 0x7fa2300757e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:39.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:39.274+0000 7fa22e59c640 1 -- 172.21.15.142:0/3429609051 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2300760a0 con 0x7fa230102760 2024-09-06T14:24:39.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:39.274+0000 7fa22ed9d640 1 --2- 172.21.15.142:0/3429609051 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa230104280 0x7fa2300757e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-06T14:24:39.274 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:39.274+0000 7fa22e59c640 1 --2- 172.21.15.142:0/3429609051 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa230102760 0x7fa230076a30 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fa220035040 tx=0x7fa22002aed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:24:39.274 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:39.274+0000 7fa21f7fe640 1 -- 172.21.15.142:0/3429609051 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa220037040 con 0x7fa230102760 2024-09-06T14:24:39.274 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:39.274+0000 7fa21f7fe640 1 -- 172.21.15.142:0/3429609051 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7fa220005070 con 0x7fa230102760 2024-09-06T14:24:39.274 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:39.274+0000 7fa234900640 1 -- 172.21.15.142:0/3429609051 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2301a2360 con 0x7fa230102760 2024-09-06T14:24:39.274 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:39.274+0000 7fa21f7fe640 1 -- 172.21.15.142:0/3429609051 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa220003830 con 0x7fa230102760 2024-09-06T14:24:39.274 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:39.274+0000 7fa234900640 1 -- 172.21.15.142:0/3429609051 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2301a2860 con 0x7fa230102760 2024-09-06T14:24:39.275 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:39.275+0000 7fa234900640 1 -- 172.21.15.142:0/3429609051 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa230102b60 con 0x7fa230102760 2024-09-06T14:24:39.275 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:39.276+0000 7fa21f7fe640 1 -- 172.21.15.142:0/3429609051 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 16) ==== 99837+0+0 (secure 0 0 0) 0x7fa220030a60 con 0x7fa230102760 2024-09-06T14:24:39.275 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:39.276+0000 7fa21f7fe640 1 --2- 172.21.15.142:0/3429609051 >> v2:172.21.15.142:6800/853729507 conn(0x7fa2180775c0 0x7fa218079a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:39.275 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:39.276+0000 7fa21f7fe640 1 -- 172.21.15.142:0/3429609051 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 5856+0+0 (secure 0 0 0) 0x7fa2200b9100 con 0x7fa230102760 2024-09-06T14:24:39.275 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:39.276+0000 7fa22dd9b640 1 --2- 172.21.15.142:0/3429609051 >> v2:172.21.15.142:6800/853729507 conn(0x7fa2180775c0 0x7fa218079a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:39.275 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:39.277+0000 7fa22dd9b640 1 --2- 172.21.15.142:0/3429609051 >> v2:172.21.15.142:6800/853729507 conn(0x7fa2180775c0 0x7fa218079a80 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fa224009bd0 tx=0x7fa224005f90 comp rx=0 tx=0).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:24:39.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:39.280+0000 7fa21f7fe640 1 -- 172.21.15.142:0/3429609051 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7fa220082280 con 0x7fa230102760 2024-09-06T14:24:39.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:38 smithi162 ceph-mon[33787]: Deploying daemon prometheus.a on smithi162 2024-09-06T14:24:39.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:38 smithi162 ceph-mon[33787]: from='client.24436 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi162=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-06T14:24:39.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:38 smithi162 ceph-mon[33787]: Saving service grafana spec with placement smithi162=a;count:1 2024-09-06T14:24:39.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:38 smithi162 ceph-mon[33787]: mgrmap e16: y(active, since 4m), standbys: x 2024-09-06T14:24:39.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:39.489+0000 7fa234900640 1 -- 172.21.15.142:0/3429609051 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7fa23002e6a0 con 0x7fa230102760 2024-09-06T14:24:39.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:39.494+0000 7fa21f7fe640 1 -- 172.21.15.142:0/3429609051 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v16) ==== 170+0+59 (secure 0 0 0) 0x7fa220081b80 con 0x7fa230102760 2024-09-06T14:24:39.491 INFO:teuthology.orchestra.run.smithi142.stdout:[client.0] 2024-09-06T14:24:39.491 INFO:teuthology.orchestra.run.smithi142.stdout: key = AQCnENtmP3tHHRAA933D0mq/t5gqokBqYipPVQ== 2024-09-06T14:24:39.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:39.496+0000 7fa234900640 1 -- 172.21.15.142:0/3429609051 >> v2:172.21.15.142:6800/853729507 conn(0x7fa2180775c0 msgr2=0x7fa218079a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:39.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:39.496+0000 7fa234900640 1 --2- 172.21.15.142:0/3429609051 >> v2:172.21.15.142:6800/853729507 conn(0x7fa2180775c0 0x7fa218079a80 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fa224009bd0 tx=0x7fa224005f90 comp rx=0 tx=0).stop 2024-09-06T14:24:39.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:39.496+0000 7fa234900640 1 -- 172.21.15.142:0/3429609051 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa230102760 msgr2=0x7fa230076a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:39.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:39.496+0000 7fa234900640 1 --2- 172.21.15.142:0/3429609051 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa230102760 0x7fa230076a30 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fa220035040 tx=0x7fa22002aed0 comp rx=0 tx=0).stop 2024-09-06T14:24:39.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:39.496+0000 7fa234900640 1 -- 172.21.15.142:0/3429609051 shutdown_connections 2024-09-06T14:24:39.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:39.496+0000 7fa234900640 1 --2- 172.21.15.142:0/3429609051 >> v2:172.21.15.142:6800/853729507 conn(0x7fa2180775c0 0x7fa218079a80 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:39.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:39.496+0000 7fa234900640 1 --2- 172.21.15.142:0/3429609051 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa230104280 0x7fa2300757e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:39.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:39.496+0000 7fa234900640 1 --2- 172.21.15.142:0/3429609051 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa2301038c0 0x7fa2300752a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:39.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:39.496+0000 7fa234900640 1 --2- 172.21.15.142:0/3429609051 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa230102760 0x7fa230076a30 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:39.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:39.496+0000 7fa234900640 1 -- 172.21.15.142:0/3429609051 >> 172.21.15.142:0/3429609051 conn(0x7fa2300fe050 msgr2=0x7fa2300ffbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:24:39.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:39.496+0000 7fa234900640 1 -- 172.21.15.142:0/3429609051 shutdown_connections 2024-09-06T14:24:39.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:39.496+0000 7fa234900640 1 -- 172.21.15.142:0/3429609051 wait complete. 2024-09-06T14:24:40.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:39 smithi142 ceph-mon[29958]: pgmap v191: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 4.9 KiB/s wr, 162 op/s 2024-09-06T14:24:40.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:39 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:40.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:39 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/3429609051' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-06T14:24:40.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:39 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/3429609051' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-06T14:24:40.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:39 smithi142 ceph-mon[37372]: pgmap v191: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 4.9 KiB/s wr, 162 op/s 2024-09-06T14:24:40.210 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:39 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:40.210 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:39 smithi142 ceph-mon[37372]: from='client.? 172.21.15.142:0/3429609051' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-06T14:24:40.210 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:39 smithi142 ceph-mon[37372]: from='client.? 172.21.15.142:0/3429609051' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-06T14:24:40.236 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-09-06T14:24:40.236 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-09-06T14:24:40.236 DEBUG:teuthology.orchestra.run.smithi142:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-06T14:24:40.273 DEBUG:teuthology.orchestra.run.smithi162:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-06T14:24:40.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:39 smithi162 ceph-mon[33787]: pgmap v191: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 4.9 KiB/s wr, 162 op/s 2024-09-06T14:24:40.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:39 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:40.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:39 smithi162 ceph-mon[33787]: from='client.? 172.21.15.142:0/3429609051' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-06T14:24:40.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:39 smithi162 ceph-mon[33787]: from='client.? 172.21.15.142:0/3429609051' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-06T14:24:40.562 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.b/config 2024-09-06T14:24:41.409 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:41.372+0000 7fcbd0f9c640 1 -- 172.21.15.162:0/2447603605 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcbcc102780 msgr2=0x7fcbcc102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:41.409 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:41.372+0000 7fcbd0f9c640 1 --2- 172.21.15.162:0/2447603605 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcbcc102780 0x7fcbcc102b80 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fcbbc009930 tx=0x7fcbbc02a420 comp rx=0 tx=0).stop 2024-09-06T14:24:41.409 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:41.372+0000 7fcbd0f9c640 1 -- 172.21.15.162:0/2447603605 shutdown_connections 2024-09-06T14:24:41.409 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:41.372+0000 7fcbd0f9c640 1 --2- 172.21.15.162:0/2447603605 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcbcc1042a0 0x7fcbcc10ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:41.409 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:41.372+0000 7fcbd0f9c640 1 --2- 172.21.15.162:0/2447603605 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcbcc1038e0 0x7fcbcc103d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:41.409 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:41.372+0000 7fcbd0f9c640 1 --2- 172.21.15.162:0/2447603605 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcbcc102780 0x7fcbcc102b80 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:41.409 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:41.372+0000 7fcbd0f9c640 1 -- 172.21.15.162:0/2447603605 >> 172.21.15.162:0/2447603605 conn(0x7fcbcc0fe050 msgr2=0x7fcbcc100490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:24:41.409 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:41.373+0000 7fcbd0f9c640 1 -- 172.21.15.162:0/2447603605 shutdown_connections 2024-09-06T14:24:41.410 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:41.373+0000 7fcbd0f9c640 1 -- 172.21.15.162:0/2447603605 wait complete. 2024-09-06T14:24:41.410 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:41.373+0000 7fcbd0f9c640 1 Processor -- start 2024-09-06T14:24:41.410 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:41.373+0000 7fcbd0f9c640 1 -- start start 2024-09-06T14:24:41.410 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:41.374+0000 7fcbd0f9c640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcbcc102780 0x7fcbcc076a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:41.410 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:41.374+0000 7fcbd0f9c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcbcc1038e0 0x7fcbcc0752a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:41.410 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:41.374+0000 7fcbd0f9c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcbcc1042a0 0x7fcbcc0757e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:41.410 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:41.374+0000 7fcbd0f9c640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fcbcc10dac0 con 0x7fcbcc1038e0 2024-09-06T14:24:41.410 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:41.374+0000 7fcbd0f9c640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fcbcc10d940 con 0x7fcbcc1042a0 2024-09-06T14:24:41.410 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:41.374+0000 7fcbd0f9c640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7fcbcc10dc40 con 0x7fcbcc102780 2024-09-06T14:24:41.410 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:41.374+0000 7fcbcad9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcbcc1042a0 0x7fcbcc0757e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:41.410 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:41.374+0000 7fcbcad9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcbcc1042a0 0x7fcbcc0757e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:39656/0 (socket says 172.21.15.162:39656) 2024-09-06T14:24:41.410 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:41.374+0000 7fcbcad9d640 1 -- 172.21.15.162:0/537066979 learned_addr learned my addr 172.21.15.162:0/537066979 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-09-06T14:24:41.410 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:41.374+0000 7fcbca59c640 1 --2- 172.21.15.162:0/537066979 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcbcc102780 0x7fcbcc076a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:41.410 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:41.374+0000 7fcbc9d9b640 1 --2- 172.21.15.162:0/537066979 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcbcc1038e0 0x7fcbcc0752a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:41.411 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:41.374+0000 7fcbcad9d640 1 -- 172.21.15.162:0/537066979 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcbcc102780 msgr2=0x7fcbcc076a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:41.411 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:41.374+0000 7fcbcad9d640 1 --2- 172.21.15.162:0/537066979 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcbcc102780 0x7fcbcc076a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:41.411 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:41.374+0000 7fcbcad9d640 1 -- 172.21.15.162:0/537066979 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcbcc1038e0 msgr2=0x7fcbcc0752a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:41.411 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:41.374+0000 7fcbcad9d640 1 --2- 172.21.15.162:0/537066979 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcbcc1038e0 0x7fcbcc0752a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:41.411 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:41.374+0000 7fcbcad9d640 1 -- 172.21.15.162:0/537066979 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbcc0760a0 con 0x7fcbcc1042a0 2024-09-06T14:24:41.411 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:41.374+0000 7fcbcad9d640 1 --2- 172.21.15.162:0/537066979 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcbcc1042a0 0x7fcbcc0757e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fcbc400eaa0 tx=0x7fcbc400ef60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:24:41.411 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:41.374+0000 7fcbbb7fe640 1 -- 172.21.15.162:0/537066979 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbc400cdb0 con 0x7fcbcc1042a0 2024-09-06T14:24:41.411 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:41.374+0000 7fcbd0f9c640 1 -- 172.21.15.162:0/537066979 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbcc1a23c0 con 0x7fcbcc1042a0 2024-09-06T14:24:41.411 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:41.374+0000 7fcbbb7fe640 1 -- 172.21.15.162:0/537066979 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7fcbc4005020 con 0x7fcbcc1042a0 2024-09-06T14:24:41.411 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:41.374+0000 7fcbbb7fe640 1 -- 172.21.15.162:0/537066979 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbc40116d0 con 0x7fcbcc1042a0 2024-09-06T14:24:41.411 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:41.375+0000 7fcbd0f9c640 1 -- 172.21.15.162:0/537066979 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbcc1a2920 con 0x7fcbcc1042a0 2024-09-06T14:24:41.411 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:41.376+0000 7fcbd0f9c640 1 -- 172.21.15.162:0/537066979 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcbcc10e280 con 0x7fcbcc1042a0 2024-09-06T14:24:41.411 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:41.376+0000 7fcbbb7fe640 1 -- 172.21.15.162:0/537066979 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 16) ==== 99837+0+0 (secure 0 0 0) 0x7fcbc4004a40 con 0x7fcbcc1042a0 2024-09-06T14:24:41.411 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:41.377+0000 7fcbbb7fe640 1 --2- 172.21.15.162:0/537066979 >> v2:172.21.15.142:6800/853729507 conn(0x7fcbb40775c0 0x7fcbb4079a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:41.412 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:41.377+0000 7fcbbb7fe640 1 -- 172.21.15.162:0/537066979 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 5856+0+0 (secure 0 0 0) 0x7fcbc4099910 con 0x7fcbcc1042a0 2024-09-06T14:24:41.412 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:41.377+0000 7fcbca59c640 1 --2- 172.21.15.162:0/537066979 >> v2:172.21.15.142:6800/853729507 conn(0x7fcbb40775c0 0x7fcbb4079a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:41.412 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:41.377+0000 7fcbca59c640 1 --2- 172.21.15.162:0/537066979 >> v2:172.21.15.142:6800/853729507 conn(0x7fcbb40775c0 0x7fcbb4079a80 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fcbbc009900 tx=0x7fcbbc000bc0 comp rx=0 tx=0).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:24:41.412 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:41.381+0000 7fcbbb7fe640 1 -- 172.21.15.162:0/537066979 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7fcbc4016030 con 0x7fcbcc1042a0 2024-09-06T14:24:41.588 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:41.591+0000 7fcbd0f9c640 1 -- 172.21.15.162:0/537066979 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7fcbcc109320 con 0x7fcbcc1042a0 2024-09-06T14:24:41.593 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:41.596+0000 7fcbbb7fe640 1 -- 172.21.15.162:0/537066979 <== mon.1 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v17) ==== 170+0+59 (secure 0 0 0) 0x7fcbc40623c0 con 0x7fcbcc1042a0 2024-09-06T14:24:41.593 INFO:teuthology.orchestra.run.smithi162.stdout:[client.1] 2024-09-06T14:24:41.593 INFO:teuthology.orchestra.run.smithi162.stdout: key = AQCpENtmPhFiIxAAejbgS7LHzoW+1zkI94MV/g== 2024-09-06T14:24:41.595 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:41.598+0000 7fcbd0f9c640 1 -- 172.21.15.162:0/537066979 >> v2:172.21.15.142:6800/853729507 conn(0x7fcbb40775c0 msgr2=0x7fcbb4079a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:41.595 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:41.598+0000 7fcbd0f9c640 1 --2- 172.21.15.162:0/537066979 >> v2:172.21.15.142:6800/853729507 conn(0x7fcbb40775c0 0x7fcbb4079a80 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fcbbc009900 tx=0x7fcbbc000bc0 comp rx=0 tx=0).stop 2024-09-06T14:24:41.595 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:41.598+0000 7fcbd0f9c640 1 -- 172.21.15.162:0/537066979 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcbcc1042a0 msgr2=0x7fcbcc0757e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:41.596 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:41.598+0000 7fcbd0f9c640 1 --2- 172.21.15.162:0/537066979 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcbcc1042a0 0x7fcbcc0757e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fcbc400eaa0 tx=0x7fcbc400ef60 comp rx=0 tx=0).stop 2024-09-06T14:24:41.596 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:41.598+0000 7fcbd0f9c640 1 -- 172.21.15.162:0/537066979 shutdown_connections 2024-09-06T14:24:41.596 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:41.598+0000 7fcbd0f9c640 1 --2- 172.21.15.162:0/537066979 >> v2:172.21.15.142:6800/853729507 conn(0x7fcbb40775c0 0x7fcbb4079a80 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:41.596 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:41.598+0000 7fcbd0f9c640 1 --2- 172.21.15.162:0/537066979 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcbcc1042a0 0x7fcbcc0757e0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:41.596 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:41.598+0000 7fcbd0f9c640 1 --2- 172.21.15.162:0/537066979 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcbcc1038e0 0x7fcbcc0752a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:41.596 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:41.598+0000 7fcbd0f9c640 1 --2- 172.21.15.162:0/537066979 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcbcc102780 0x7fcbcc076a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:41.596 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:41.598+0000 7fcbd0f9c640 1 -- 172.21.15.162:0/537066979 >> 172.21.15.162:0/537066979 conn(0x7fcbcc0fe050 msgr2=0x7fcbcc0ffbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:24:41.596 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:41.598+0000 7fcbd0f9c640 1 -- 172.21.15.162:0/537066979 shutdown_connections 2024-09-06T14:24:41.596 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-06T14:24:41.598+0000 7fcbd0f9c640 1 -- 172.21.15.162:0/537066979 wait complete. 2024-09-06T14:24:42.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:41 smithi142 ceph-mon[29958]: pgmap v192: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s rd, 4.2 KiB/s wr, 139 op/s 2024-09-06T14:24:42.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:41 smithi142 ceph-mon[29958]: from='client.? 172.21.15.162:0/537066979' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-06T14:24:42.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:41 smithi142 ceph-mon[29958]: from='client.? ' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-06T14:24:42.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:41 smithi142 ceph-mon[29958]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-06T14:24:42.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:41 smithi142 ceph-mon[37372]: pgmap v192: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s rd, 4.2 KiB/s wr, 139 op/s 2024-09-06T14:24:42.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:41 smithi142 ceph-mon[37372]: from='client.? 172.21.15.162:0/537066979' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-06T14:24:42.210 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:41 smithi142 ceph-mon[37372]: from='client.? ' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-06T14:24:42.210 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:41 smithi142 ceph-mon[37372]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-06T14:24:42.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:41 smithi162 ceph-mon[33787]: pgmap v192: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s rd, 4.2 KiB/s wr, 139 op/s 2024-09-06T14:24:42.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:41 smithi162 ceph-mon[33787]: from='client.? 172.21.15.162:0/537066979' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-06T14:24:42.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:41 smithi162 ceph-mon[33787]: from='client.? ' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-06T14:24:42.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:41 smithi162 ceph-mon[33787]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-06T14:24:43.899 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-09-06T14:24:43.899 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-09-06T14:24:43.900 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-09-06T14:24:43.979 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph config log 1 --format=json 2024-09-06T14:24:44.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:43 smithi142 ceph-mon[29958]: pgmap v193: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 3.5 KiB/s wr, 114 op/s 2024-09-06T14:24:44.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:43 smithi142 ceph-mon[37372]: pgmap v193: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 3.5 KiB/s wr, 114 op/s 2024-09-06T14:24:44.242 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:24:44.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:43 smithi162 ceph-mon[33787]: pgmap v193: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 3.5 KiB/s wr, 114 op/s 2024-09-06T14:24:44.710 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:44.703+0000 7fdaec4ae640 1 -- 172.21.15.142:0/2586507817 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdae41019c0 msgr2=0x7fdae4101da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:44.710 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:44.703+0000 7fdaec4ae640 1 --2- 172.21.15.142:0/2586507817 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdae41019c0 0x7fdae4101da0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fdae0009900 tx=0x7fdae002a410 comp rx=0 tx=0).stop 2024-09-06T14:24:44.710 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:44.704+0000 7fdaec4ae640 1 -- 172.21.15.142:0/2586507817 shutdown_connections 2024-09-06T14:24:44.710 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:44.704+0000 7fdaec4ae640 1 --2- 172.21.15.142:0/2586507817 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdae4107d50 0x7fdae410e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:44.710 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:44.704+0000 7fdaec4ae640 1 --2- 172.21.15.142:0/2586507817 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdae41073d0 0x7fdae4107810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:44.711 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:44.704+0000 7fdaec4ae640 1 --2- 172.21.15.142:0/2586507817 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdae41019c0 0x7fdae4101da0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:44.711 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:44.704+0000 7fdaec4ae640 1 -- 172.21.15.142:0/2586507817 >> 172.21.15.142:0/2586507817 conn(0x7fdae40fd430 msgr2=0x7fdae40ff850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:24:44.711 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:44.704+0000 7fdaec4ae640 1 -- 172.21.15.142:0/2586507817 shutdown_connections 2024-09-06T14:24:44.711 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:44.704+0000 7fdaec4ae640 1 -- 172.21.15.142:0/2586507817 wait complete. 2024-09-06T14:24:44.711 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:44.704+0000 7fdaec4ae640 1 Processor -- start 2024-09-06T14:24:44.711 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:44.704+0000 7fdaec4ae640 1 -- start start 2024-09-06T14:24:44.711 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:44.704+0000 7fdaec4ae640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdae41019c0 0x7fdae4198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:44.711 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:44.704+0000 7fdaec4ae640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdae41073d0 0x7fdae4198c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:44.711 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:44.704+0000 7fdaec4ae640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdae4107d50 0x7fdae419e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:44.711 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:44.704+0000 7fdaec4ae640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fdae4111120 con 0x7fdae41073d0 2024-09-06T14:24:44.711 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:44.704+0000 7fdaec4ae640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fdae4110fa0 con 0x7fdae41019c0 2024-09-06T14:24:44.711 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:44.704+0000 7fdaec4ae640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7fdae41112a0 con 0x7fdae4107d50 2024-09-06T14:24:44.711 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:44.705+0000 7fdae9a49640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdae41073d0 0x7fdae4198c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:44.712 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:44.705+0000 7fdaeaa4b640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdae4107d50 0x7fdae419e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:44.712 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:44.705+0000 7fdaea24a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdae41019c0 0x7fdae4198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:44.712 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:44.705+0000 7fdaeaa4b640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdae4107d50 0x7fdae419e010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:53278/0 (socket says 172.21.15.142:53278) 2024-09-06T14:24:44.712 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:44.705+0000 7fdaea24a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdae41019c0 0x7fdae4198730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.142:45694/0 (socket says 172.21.15.142:45694) 2024-09-06T14:24:44.712 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:44.705+0000 7fdaeaa4b640 1 -- 172.21.15.142:0/4038124314 learned_addr learned my addr 172.21.15.142:0/4038124314 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:24:44.712 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:44.705+0000 7fdaeaa4b640 1 -- 172.21.15.142:0/4038124314 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdae41019c0 msgr2=0x7fdae4198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:44.712 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:44.705+0000 7fdaeaa4b640 1 --2- 172.21.15.142:0/4038124314 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdae41019c0 0x7fdae4198730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:44.712 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:44.705+0000 7fdaeaa4b640 1 -- 172.21.15.142:0/4038124314 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdae41073d0 msgr2=0x7fdae4198c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:44.712 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:44.705+0000 7fdaeaa4b640 1 --2- 172.21.15.142:0/4038124314 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdae41073d0 0x7fdae4198c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:44.712 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:44.705+0000 7fdaeaa4b640 1 -- 172.21.15.142:0/4038124314 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdae419e7e0 con 0x7fdae4107d50 2024-09-06T14:24:44.712 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:44.705+0000 7fdae9a49640 1 --2- 172.21.15.142:0/4038124314 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdae41073d0 0x7fdae4198c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-06T14:24:44.712 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:44.705+0000 7fdaeaa4b640 1 --2- 172.21.15.142:0/4038124314 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdae4107d50 0x7fdae419e010 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fdadc00eb10 tx=0x7fdadc00c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:24:44.712 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:44.705+0000 7fdadb7fe640 1 -- 172.21.15.142:0/4038124314 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdadc004b20 con 0x7fdae4107d50 2024-09-06T14:24:44.712 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:44.705+0000 7fdadb7fe640 1 -- 172.21.15.142:0/4038124314 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7fdadc004cc0 con 0x7fdae4107d50 2024-09-06T14:24:44.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:44.705+0000 7fdaec4ae640 1 -- 172.21.15.142:0/4038124314 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdae419ead0 con 0x7fdae4107d50 2024-09-06T14:24:44.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:44.705+0000 7fdadb7fe640 1 -- 172.21.15.142:0/4038124314 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdadc010690 con 0x7fdae4107d50 2024-09-06T14:24:44.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:44.705+0000 7fdaec4ae640 1 -- 172.21.15.142:0/4038124314 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fdae41a5190 con 0x7fdae4107d50 2024-09-06T14:24:44.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:44.706+0000 7fdaec4ae640 1 -- 172.21.15.142:0/4038124314 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdae4101da0 con 0x7fdae4107d50 2024-09-06T14:24:44.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:44.707+0000 7fdadb7fe640 1 -- 172.21.15.142:0/4038124314 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 16) ==== 99837+0+0 (secure 0 0 0) 0x7fdadc005020 con 0x7fdae4107d50 2024-09-06T14:24:44.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:44.707+0000 7fdadb7fe640 1 --2- 172.21.15.142:0/4038124314 >> v2:172.21.15.142:6800/853729507 conn(0x7fdad00775c0 0x7fdad0079a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:44.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:44.708+0000 7fdadb7fe640 1 -- 172.21.15.142:0/4038124314 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(61..61 src has 1..61) ==== 5856+0+0 (secure 0 0 0) 0x7fdadc099cd0 con 0x7fdae4107d50 2024-09-06T14:24:44.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:44.708+0000 7fdaea24a640 1 --2- 172.21.15.142:0/4038124314 >> v2:172.21.15.142:6800/853729507 conn(0x7fdad00775c0 0x7fdad0079a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:44.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:44.708+0000 7fdaea24a640 1 --2- 172.21.15.142:0/4038124314 >> v2:172.21.15.142:6800/853729507 conn(0x7fdad00775c0 0x7fdad0079a80 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fdae0031440 tx=0x7fdae0005b50 comp rx=0 tx=0).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:24:44.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:44.712+0000 7fdadb7fe640 1 -- 172.21.15.142:0/4038124314 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7fdadc062780 con 0x7fdae4107d50 2024-09-06T14:24:44.865 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:44.869+0000 7fdaec4ae640 1 -- 172.21.15.142:0/4038124314 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "config log", "num": 1, "format": "json"} v 0) -- 0x7fdae410ab40 con 0x7fdae4107d50 2024-09-06T14:24:44.866 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:44.869+0000 7fdadb7fe640 1 -- 172.21.15.142:0/4038124314 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "config log", "num": 1, "format": "json"}]=0 v24) ==== 86+0+187 (secure 0 0 0) 0x7fdadc066f40 con 0x7fdae4107d50 2024-09-06T14:24:44.866 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:24:44.868 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:44.871+0000 7fdaec4ae640 1 -- 172.21.15.142:0/4038124314 >> v2:172.21.15.142:6800/853729507 conn(0x7fdad00775c0 msgr2=0x7fdad0079a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:44.869 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:44.871+0000 7fdaec4ae640 1 --2- 172.21.15.142:0/4038124314 >> v2:172.21.15.142:6800/853729507 conn(0x7fdad00775c0 0x7fdad0079a80 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fdae0031440 tx=0x7fdae0005b50 comp rx=0 tx=0).stop 2024-09-06T14:24:44.869 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:44.871+0000 7fdaec4ae640 1 -- 172.21.15.142:0/4038124314 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdae4107d50 msgr2=0x7fdae419e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:44.869 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:44.871+0000 7fdaec4ae640 1 --2- 172.21.15.142:0/4038124314 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdae4107d50 0x7fdae419e010 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fdadc00eb10 tx=0x7fdadc00c2d0 comp rx=0 tx=0).stop 2024-09-06T14:24:44.869 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:44.871+0000 7fdaec4ae640 1 -- 172.21.15.142:0/4038124314 shutdown_connections 2024-09-06T14:24:44.869 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:44.871+0000 7fdaec4ae640 1 --2- 172.21.15.142:0/4038124314 >> v2:172.21.15.142:6800/853729507 conn(0x7fdad00775c0 0x7fdad0079a80 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:44.869 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:44.871+0000 7fdaec4ae640 1 --2- 172.21.15.142:0/4038124314 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdae4107d50 0x7fdae419e010 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:44.870 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:44.871+0000 7fdaec4ae640 1 --2- 172.21.15.142:0/4038124314 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdae41073d0 0x7fdae4198c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:44.870 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:44.871+0000 7fdaec4ae640 1 --2- 172.21.15.142:0/4038124314 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdae41019c0 0x7fdae4198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:44.870 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:44.871+0000 7fdaec4ae640 1 -- 172.21.15.142:0/4038124314 >> 172.21.15.142:0/4038124314 conn(0x7fdae40fd430 msgr2=0x7fdae406aac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:24:44.870 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:44.872+0000 7fdaec4ae640 1 -- 172.21.15.142:0/4038124314 shutdown_connections 2024-09-06T14:24:44.870 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:44.872+0000 7fdaec4ae640 1 -- 172.21.15.142:0/4038124314 wait complete. 2024-09-06T14:24:45.458 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:45 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:45.459 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:45 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/4038124314' entity='client.admin' cmd={"prefix": "config log", "num": 1, "format": "json"} : dispatch 2024-09-06T14:24:45.459 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:45 smithi142 ceph-mon[29958]: pgmap v194: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 3.2 KiB/s wr, 102 op/s 2024-09-06T14:24:45.459 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:45 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:45.459 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:45 smithi142 ceph-mon[37372]: from='client.? 172.21.15.142:0/4038124314' entity='client.admin' cmd={"prefix": "config log", "num": 1, "format": "json"} : dispatch 2024-09-06T14:24:45.459 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:45 smithi142 ceph-mon[37372]: pgmap v194: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 3.2 KiB/s wr, 102 op/s 2024-09-06T14:24:45.547 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:45 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:45.547 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:45 smithi162 ceph-mon[33787]: from='client.? 172.21.15.142:0/4038124314' entity='client.admin' cmd={"prefix": "config log", "num": 1, "format": "json"} : dispatch 2024-09-06T14:24:45.547 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:45 smithi162 ceph-mon[33787]: pgmap v194: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 3.2 KiB/s wr, 102 op/s 2024-09-06T14:24:45.769 INFO:teuthology.orchestra.run.smithi142.stdout:[{"version":24,"timestamp":"2024-09-06T14:24:33.244679+0000","name":"","changes":[{"name":"mgr/mgr/dashboard/RGW_API_SECRET_KEY","new_value":"mELwBQROKAIWeqlcvtqgQqqg5vaN8a8Tf3ZgokGH"}]}] 2024-09-06T14:24:45.770 INFO:tasks.ceph_manager:config epoch is 24 2024-09-06T14:24:45.770 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-06T14:24:45.771 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-09-06T14:24:45.771 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph mgr dump --format=json 2024-09-06T14:24:46.030 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:24:46.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:46.560+0000 7f253cd8f640 1 -- 172.21.15.142:0/4228777773 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f25381019e0 msgr2=0x7f2538101dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:46.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:46.560+0000 7f253cd8f640 1 --2- 172.21.15.142:0/4228777773 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f25381019e0 0x7f2538101dc0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f2530009860 tx=0x7f253002a340 comp rx=0 tx=0).stop 2024-09-06T14:24:46.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:46.560+0000 7f253cd8f640 1 -- 172.21.15.142:0/4228777773 shutdown_connections 2024-09-06T14:24:46.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:46.560+0000 7f253cd8f640 1 --2- 172.21.15.142:0/4228777773 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2538107d70 0x7f253810e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:46.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:46.560+0000 7f253cd8f640 1 --2- 172.21.15.142:0/4228777773 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f25381073f0 0x7f2538107830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:46.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:46.560+0000 7f253cd8f640 1 --2- 172.21.15.142:0/4228777773 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f25381019e0 0x7f2538101dc0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:46.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:46.560+0000 7f253cd8f640 1 -- 172.21.15.142:0/4228777773 >> 172.21.15.142:0/4228777773 conn(0x7f25380fd430 msgr2=0x7f25380ff850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:24:46.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:46.561+0000 7f253cd8f640 1 -- 172.21.15.142:0/4228777773 shutdown_connections 2024-09-06T14:24:46.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:46.561+0000 7f253cd8f640 1 -- 172.21.15.142:0/4228777773 wait complete. 2024-09-06T14:24:46.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:46.561+0000 7f253cd8f640 1 Processor -- start 2024-09-06T14:24:46.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:46.561+0000 7f253cd8f640 1 -- start start 2024-09-06T14:24:46.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:46.562+0000 7f253cd8f640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f25381019e0 0x7f25381975f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:46.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:46.562+0000 7f253cd8f640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f25381073f0 0x7f2538197b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:46.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:46.562+0000 7f253cd8f640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2538107d70 0x7f253819ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:46.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:46.562+0000 7f253cd8f640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f2538111100 con 0x7f25381073f0 2024-09-06T14:24:46.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:46.562+0000 7f253cd8f640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f2538110f80 con 0x7f25381019e0 2024-09-06T14:24:46.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:46.562+0000 7f253cd8f640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7f2538111280 con 0x7f2538107d70 2024-09-06T14:24:46.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:46.562+0000 7f2535d9b640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f25381073f0 0x7f2538197b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:46.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:46.562+0000 7f2536d9d640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2538107d70 0x7f253819ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:46.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:46.562+0000 7f2535d9b640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f25381073f0 0x7f2538197b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:43396/0 (socket says 172.21.15.142:43396) 2024-09-06T14:24:46.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:46.562+0000 7f2535d9b640 1 -- 172.21.15.142:0/2844943935 learned_addr learned my addr 172.21.15.142:0/2844943935 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:24:46.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:46.562+0000 7f253659c640 1 --2- 172.21.15.142:0/2844943935 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f25381019e0 0x7f25381975f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:46.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:46.562+0000 7f2536d9d640 1 -- 172.21.15.142:0/2844943935 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f25381019e0 msgr2=0x7f25381975f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:46.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:46.562+0000 7f2536d9d640 1 --2- 172.21.15.142:0/2844943935 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f25381019e0 0x7f25381975f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:46.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:46.562+0000 7f2536d9d640 1 -- 172.21.15.142:0/2844943935 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f25381073f0 msgr2=0x7f2538197b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:46.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:46.562+0000 7f2536d9d640 1 --2- 172.21.15.142:0/2844943935 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f25381073f0 0x7f2538197b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:46.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:46.562+0000 7f2536d9d640 1 -- 172.21.15.142:0/2844943935 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f253819d6a0 con 0x7f2538107d70 2024-09-06T14:24:46.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:46.562+0000 7f2535d9b640 1 --2- 172.21.15.142:0/2844943935 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f25381073f0 0x7f2538197b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-06T14:24:46.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:46.562+0000 7f2536d9d640 1 --2- 172.21.15.142:0/2844943935 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2538107d70 0x7f253819ced0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f252800d6b0 tx=0x7f252800db70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:24:46.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:46.562+0000 7f252f7fe640 1 -- 172.21.15.142:0/2844943935 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2528017070 con 0x7f2538107d70 2024-09-06T14:24:46.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:46.563+0000 7f252f7fe640 1 -- 172.21.15.142:0/2844943935 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7f2528009ca0 con 0x7f2538107d70 2024-09-06T14:24:46.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:46.563+0000 7f253cd8f640 1 -- 172.21.15.142:0/2844943935 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f253819d990 con 0x7f2538107d70 2024-09-06T14:24:46.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:46.563+0000 7f252f7fe640 1 -- 172.21.15.142:0/2844943935 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2528003bc0 con 0x7f2538107d70 2024-09-06T14:24:46.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:46.563+0000 7f253cd8f640 1 -- 172.21.15.142:0/2844943935 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f25381a4050 con 0x7f2538107d70 2024-09-06T14:24:46.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:46.564+0000 7f253cd8f640 1 -- 172.21.15.142:0/2844943935 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2538101dc0 con 0x7f2538107d70 2024-09-06T14:24:46.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:46.564+0000 7f252f7fe640 1 -- 172.21.15.142:0/2844943935 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 16) ==== 99837+0+0 (secure 0 0 0) 0x7f2528009820 con 0x7f2538107d70 2024-09-06T14:24:46.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:46.565+0000 7f252f7fe640 1 --2- 172.21.15.142:0/2844943935 >> v2:172.21.15.142:6800/853729507 conn(0x7f25240775c0 0x7f2524079a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:46.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:46.565+0000 7f253659c640 1 --2- 172.21.15.142:0/2844943935 >> v2:172.21.15.142:6800/853729507 conn(0x7f25240775c0 0x7f2524079a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:46.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:46.565+0000 7f252f7fe640 1 -- 172.21.15.142:0/2844943935 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(61..61 src has 1..61) ==== 5856+0+0 (secure 0 0 0) 0x7f2528099020 con 0x7f2538107d70 2024-09-06T14:24:46.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:46.565+0000 7f253659c640 1 --2- 172.21.15.142:0/2844943935 >> v2:172.21.15.142:6800/853729507 conn(0x7f25240775c0 0x7f2524079a80 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f253002a930 tx=0x7f25300058d0 comp rx=0 tx=0).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:24:46.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:46.569+0000 7f252f7fe640 1 -- 172.21.15.142:0/2844943935 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7f2528061ad0 con 0x7f2538107d70 2024-09-06T14:24:46.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:46.764+0000 7f253cd8f640 1 -- 172.21.15.142:0/2844943935 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f2538109b50 con 0x7f2538107d70 2024-09-06T14:24:46.764 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:46.768+0000 7f252f7fe640 1 -- 172.21.15.142:0/2844943935 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v16) ==== 74+0+191855 (secure 0 0 0) 0x7f2528066290 con 0x7f2538107d70 2024-09-06T14:24:46.768 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:24:46.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:46.771+0000 7f253cd8f640 1 -- 172.21.15.142:0/2844943935 >> v2:172.21.15.142:6800/853729507 conn(0x7f25240775c0 msgr2=0x7f2524079a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:46.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:46.771+0000 7f253cd8f640 1 --2- 172.21.15.142:0/2844943935 >> v2:172.21.15.142:6800/853729507 conn(0x7f25240775c0 0x7f2524079a80 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f253002a930 tx=0x7f25300058d0 comp rx=0 tx=0).stop 2024-09-06T14:24:46.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:46.771+0000 7f253cd8f640 1 -- 172.21.15.142:0/2844943935 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2538107d70 msgr2=0x7f253819ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:46.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:46.771+0000 7f253cd8f640 1 --2- 172.21.15.142:0/2844943935 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2538107d70 0x7f253819ced0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f252800d6b0 tx=0x7f252800db70 comp rx=0 tx=0).stop 2024-09-06T14:24:46.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:46.771+0000 7f253cd8f640 1 -- 172.21.15.142:0/2844943935 shutdown_connections 2024-09-06T14:24:46.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:46.771+0000 7f253cd8f640 1 --2- 172.21.15.142:0/2844943935 >> v2:172.21.15.142:6800/853729507 conn(0x7f25240775c0 0x7f2524079a80 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:46.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:46.771+0000 7f253cd8f640 1 --2- 172.21.15.142:0/2844943935 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2538107d70 0x7f253819ced0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:46.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:46.771+0000 7f253cd8f640 1 --2- 172.21.15.142:0/2844943935 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f25381073f0 0x7f2538197b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:46.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:46.771+0000 7f253cd8f640 1 --2- 172.21.15.142:0/2844943935 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f25381019e0 0x7f25381975f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:46.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:46.771+0000 7f253cd8f640 1 -- 172.21.15.142:0/2844943935 >> 172.21.15.142:0/2844943935 conn(0x7f25380fd430 msgr2=0x7f25380ff820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:24:46.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:46.771+0000 7f253cd8f640 1 -- 172.21.15.142:0/2844943935 shutdown_connections 2024-09-06T14:24:46.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:46.771+0000 7f253cd8f640 1 -- 172.21.15.142:0/2844943935 wait complete. 2024-09-06T14:24:46.941 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:46 smithi162 ceph-mon[33787]: from='client.? 172.21.15.142:0/2844943935' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2024-09-06T14:24:47.155 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:46 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/2844943935' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2024-09-06T14:24:47.155 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:46 smithi142 ceph-mon[37372]: from='client.? 172.21.15.142:0/2844943935' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2024-09-06T14:24:47.197 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 06 14:24:46 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-iscsi-iscsi-a[59853]: debug there is no tcmu-runner data available 2024-09-06T14:24:47.579 INFO:teuthology.orchestra.run.smithi142.stdout:{"epoch":16,"flags":0,"active_gid":14146,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6800","nonce":853729507}]},"active_addr":"172.21.15.142:6800/853729507","active_change":"2024-09-06T14:19:58.842164+0000","active_mgr_features":4540719139924082687,"available":true,"standbys":[{"gid":24112,"name":"x","mgr_features":4540719139924082687,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:1.26.1","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":false,"error_string":"No module named 'ceph.fs'","module_options":{}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:1.26.1","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":false,"error_string":"No module named 'ceph.fs'","module_options":{}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.142:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"cephadm","addrvec":[{"type":"v2","addr":"172.21.15.142:0","nonce":1320572171}]},{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.142:0","nonce":2370685741}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.142:0","nonce":4044241282}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.142:0","nonce":3584421883}]}]} 2024-09-06T14:24:47.585 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-09-06T14:24:47.586 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-06T14:24:47.586 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph osd dump --format=json 2024-09-06T14:24:47.835 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:24:48.084 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:47 smithi142 ceph-mon[29958]: pgmap v195: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 39 KiB/s rd, 2.9 KiB/s wr, 94 op/s 2024-09-06T14:24:48.084 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:47 smithi142 ceph-mon[37372]: pgmap v195: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 39 KiB/s rd, 2.9 KiB/s wr, 94 op/s 2024-09-06T14:24:48.234 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:47 smithi162 ceph-mon[33787]: pgmap v195: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 39 KiB/s rd, 2.9 KiB/s wr, 94 op/s 2024-09-06T14:24:48.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:48.301+0000 7f4c62410640 1 -- 172.21.15.142:0/3461943271 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4c5c10f110 msgr2=0x7f4c5c111500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:48.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:48.301+0000 7f4c62410640 1 --2- 172.21.15.142:0/3461943271 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4c5c10f110 0x7f4c5c111500 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4c54009be0 tx=0x7f4c5402a6c0 comp rx=0 tx=0).stop 2024-09-06T14:24:48.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:48.301+0000 7f4c62410640 1 -- 172.21.15.142:0/3461943271 shutdown_connections 2024-09-06T14:24:48.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:48.301+0000 7f4c62410640 1 --2- 172.21.15.142:0/3461943271 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4c5c10f110 0x7f4c5c111500 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:48.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:48.301+0000 7f4c62410640 1 --2- 172.21.15.142:0/3461943271 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4c5c075b60 0x7f4c5c10ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:48.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:48.301+0000 7f4c62410640 1 --2- 172.21.15.142:0/3461943271 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4c5c075240 0x7f4c5c075620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:48.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:48.301+0000 7f4c62410640 1 -- 172.21.15.142:0/3461943271 >> 172.21.15.142:0/3461943271 conn(0x7f4c5c0fe570 msgr2=0x7f4c5c100990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:24:48.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:48.301+0000 7f4c62410640 1 -- 172.21.15.142:0/3461943271 shutdown_connections 2024-09-06T14:24:48.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:48.301+0000 7f4c62410640 1 -- 172.21.15.142:0/3461943271 wait complete. 2024-09-06T14:24:48.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:48.301+0000 7f4c62410640 1 Processor -- start 2024-09-06T14:24:48.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:48.302+0000 7f4c62410640 1 -- start start 2024-09-06T14:24:48.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:48.302+0000 7f4c62410640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4c5c075240 0x7f4c5c19a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:48.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:48.302+0000 7f4c62410640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4c5c075b60 0x7f4c5c19ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:48.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:48.302+0000 7f4c62410640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4c5c10f110 0x7f4c5c1a0200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:48.303 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:48.302+0000 7f4c62410640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f4c5c114400 con 0x7f4c5c075b60 2024-09-06T14:24:48.303 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:48.302+0000 7f4c62410640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f4c5c114280 con 0x7f4c5c075240 2024-09-06T14:24:48.303 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:48.302+0000 7f4c62410640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7f4c5c114580 con 0x7f4c5c10f110 2024-09-06T14:24:48.303 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:48.302+0000 7f4c609ad640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4c5c10f110 0x7f4c5c1a0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:48.303 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:48.302+0000 7f4c53fff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4c5c075b60 0x7f4c5c19ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:48.303 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:48.302+0000 7f4c609ad640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4c5c10f110 0x7f4c5c1a0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:43556/0 (socket says 172.21.15.142:43556) 2024-09-06T14:24:48.303 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:48.302+0000 7f4c53fff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4c5c075b60 0x7f4c5c19ae60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:37748/0 (socket says 172.21.15.142:37748) 2024-09-06T14:24:48.303 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:48.302+0000 7f4c609ad640 1 -- 172.21.15.142:0/3613057369 learned_addr learned my addr 172.21.15.142:0/3613057369 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:24:48.303 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:48.302+0000 7f4c5bfff640 1 --2- 172.21.15.142:0/3613057369 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4c5c075240 0x7f4c5c19a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:48.303 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:48.302+0000 7f4c609ad640 1 -- 172.21.15.142:0/3613057369 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4c5c075240 msgr2=0x7f4c5c19a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:48.303 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:48.302+0000 7f4c609ad640 1 --2- 172.21.15.142:0/3613057369 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4c5c075240 0x7f4c5c19a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:48.303 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:48.302+0000 7f4c609ad640 1 -- 172.21.15.142:0/3613057369 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4c5c075b60 msgr2=0x7f4c5c19ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:48.303 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:48.302+0000 7f4c609ad640 1 --2- 172.21.15.142:0/3613057369 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4c5c075b60 0x7f4c5c19ae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:48.303 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:48.302+0000 7f4c609ad640 1 -- 172.21.15.142:0/3613057369 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c5c1a09d0 con 0x7f4c5c10f110 2024-09-06T14:24:48.303 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:48.302+0000 7f4c53fff640 1 --2- 172.21.15.142:0/3613057369 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4c5c075b60 0x7f4c5c19ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-06T14:24:48.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:48.302+0000 7f4c609ad640 1 --2- 172.21.15.142:0/3613057369 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4c5c10f110 0x7f4c5c1a0200 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4c5402abe0 tx=0x7f4c540305d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:24:48.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:48.302+0000 7f4c59ffb640 1 -- 172.21.15.142:0/3613057369 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c54041020 con 0x7f4c5c10f110 2024-09-06T14:24:48.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:48.303+0000 7f4c59ffb640 1 -- 172.21.15.142:0/3613057369 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7f4c54030740 con 0x7f4c5c10f110 2024-09-06T14:24:48.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:48.303+0000 7f4c62410640 1 -- 172.21.15.142:0/3613057369 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c5c1a0c60 con 0x7f4c5c10f110 2024-09-06T14:24:48.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:48.303+0000 7f4c59ffb640 1 -- 172.21.15.142:0/3613057369 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c54036040 con 0x7f4c5c10f110 2024-09-06T14:24:48.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:48.303+0000 7f4c62410640 1 -- 172.21.15.142:0/3613057369 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c5c1a7380 con 0x7f4c5c10f110 2024-09-06T14:24:48.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:48.304+0000 7f4c62410640 1 -- 172.21.15.142:0/3613057369 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c5c075620 con 0x7f4c5c10f110 2024-09-06T14:24:48.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:48.304+0000 7f4c59ffb640 1 -- 172.21.15.142:0/3613057369 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 16) ==== 99837+0+0 (secure 0 0 0) 0x7f4c54044020 con 0x7f4c5c10f110 2024-09-06T14:24:48.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:48.305+0000 7f4c59ffb640 1 --2- 172.21.15.142:0/3613057369 >> v2:172.21.15.142:6800/853729507 conn(0x7f4c480775c0 0x7f4c48079a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:48.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:48.305+0000 7f4c59ffb640 1 -- 172.21.15.142:0/3613057369 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(61..61 src has 1..61) ==== 5856+0+0 (secure 0 0 0) 0x7f4c540ba2b0 con 0x7f4c5c10f110 2024-09-06T14:24:48.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:48.305+0000 7f4c5bfff640 1 --2- 172.21.15.142:0/3613057369 >> v2:172.21.15.142:6800/853729507 conn(0x7f4c480775c0 0x7f4c48079a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:48.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:48.305+0000 7f4c5bfff640 1 --2- 172.21.15.142:0/3613057369 >> v2:172.21.15.142:6800/853729507 conn(0x7f4c480775c0 0x7f4c48079a80 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c009990 tx=0x7f4c4c009110 comp rx=0 tx=0).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:24:48.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:48.309+0000 7f4c59ffb640 1 -- 172.21.15.142:0/3613057369 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7f4c5404b020 con 0x7f4c5c10f110 2024-09-06T14:24:48.456 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:48.459+0000 7f4c62410640 1 -- 172.21.15.142:0/3613057369 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4c5c1099a0 con 0x7f4c5c10f110 2024-09-06T14:24:48.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:48.461+0000 7f4c59ffb640 1 -- 172.21.15.142:0/3613057369 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v61) ==== 74+0+20638 (secure 0 0 0) 0x7f4c54083430 con 0x7f4c5c10f110 2024-09-06T14:24:48.458 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:24:48.458 INFO:teuthology.orchestra.run.smithi142.stdout:{"epoch":61,"fsid":"ce32462e-6c5a-11ef-bce4-c7b262605968","created":"2024-09-06T14:19:13.103278+0000","modified":"2024-09-06T14:24:33.877886+0000","last_up_change":"2024-09-06T14:24:16.577635+0000","last_in_change":"2024-09-06T14:23:56.281210+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-06T14:22:11.149139+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-09-06T14:24:23.519434+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":55,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-09-06T14:24:24.425747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-09-06T14:24:26.069368+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-09-06T14:24:28.014662+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-09-06T14:24:30.077167+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7823801-9043-4aaf-a680-046c5b16f18b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6801","nonce":1649056378}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":1649056378}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":1649056378}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6803","nonce":1649056378}]},"public_addr":"172.21.15.142:6801/1649056378","cluster_addr":"172.21.15.142:6802/1649056378","heartbeat_back_addr":"172.21.15.142:6804/1649056378","heartbeat_front_addr":"172.21.15.142:6803/1649056378","state":["exists","up"]},{"osd":1,"uuid":"364b1ffb-bdda-4b70-bd6c-3bcb73110359","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6805","nonce":599802221}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":599802221}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":599802221}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6807","nonce":599802221}]},"public_addr":"172.21.15.142:6805/599802221","cluster_addr":"172.21.15.142:6806/599802221","heartbeat_back_addr":"172.21.15.142:6808/599802221","heartbeat_front_addr":"172.21.15.142:6807/599802221","state":["exists","up"]},{"osd":2,"uuid":"d7b4c863-74aa-476b-9377-8262e2c77ae3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6809","nonce":1302270091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":1302270091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":1302270091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6811","nonce":1302270091}]},"public_addr":"172.21.15.142:6809/1302270091","cluster_addr":"172.21.15.142:6810/1302270091","heartbeat_back_addr":"172.21.15.142:6812/1302270091","heartbeat_front_addr":"172.21.15.142:6811/1302270091","state":["exists","up"]},{"osd":3,"uuid":"d8080de9-24b0-4d10-b745-ec2a23f80e49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6813","nonce":2917907718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":2917907718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":2917907718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6815","nonce":2917907718}]},"public_addr":"172.21.15.142:6813/2917907718","cluster_addr":"172.21.15.142:6814/2917907718","heartbeat_back_addr":"172.21.15.142:6816/2917907718","heartbeat_front_addr":"172.21.15.142:6815/2917907718","state":["exists","up"]},{"osd":4,"uuid":"d8c6b71c-01ef-44a5-b78b-49ecce11f357","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":2665827777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6801","nonce":2665827777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6803","nonce":2665827777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":2665827777}]},"public_addr":"172.21.15.162:6800/2665827777","cluster_addr":"172.21.15.162:6801/2665827777","heartbeat_back_addr":"172.21.15.162:6803/2665827777","heartbeat_front_addr":"172.21.15.162:6802/2665827777","state":["exists","up"]},{"osd":5,"uuid":"9215d7b7-c268-44bb-b688-74ab1576d1c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":3404139106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6805","nonce":3404139106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6807","nonce":3404139106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":3404139106}]},"public_addr":"172.21.15.162:6804/3404139106","cluster_addr":"172.21.15.162:6805/3404139106","heartbeat_back_addr":"172.21.15.162:6807/3404139106","heartbeat_front_addr":"172.21.15.162:6806/3404139106","state":["exists","up"]},{"osd":6,"uuid":"457940ad-3624-4613-98ce-817baa5f2d1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1155174053}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6809","nonce":1155174053}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6811","nonce":1155174053}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1155174053}]},"public_addr":"172.21.15.162:6808/1155174053","cluster_addr":"172.21.15.162:6809/1155174053","heartbeat_back_addr":"172.21.15.162:6811/1155174053","heartbeat_front_addr":"172.21.15.162:6810/1155174053","state":["exists","up"]},{"osd":7,"uuid":"b92b364c-849e-4971-be37-6c3f440e7cec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1194583876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6813","nonce":1194583876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6815","nonce":1194583876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1194583876}]},"public_addr":"172.21.15.162:6812/1194583876","cluster_addr":"172.21.15.162:6813/1194583876","heartbeat_back_addr":"172.21.15.162:6815/1194583876","heartbeat_front_addr":"172.21.15.162:6814/1194583876","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-06T14:21:14.998819+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-06T14:21:42.692114+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-06T14:22:09.477210+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-06T14:22:35.347799+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-06T14:22:58.792921+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-06T14:23:23.876765+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-06T14:23:49.484604+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-06T14:24:15.163203+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.142:0/516816086":"2024-09-07T14:19:58.842070+0000","172.21.15.142:6800/3892080990":"2024-09-07T14:19:58.842070+0000","172.21.15.142:0/3098992975":"2024-09-07T14:19:58.842070+0000","172.21.15.142:0/3489797750":"2024-09-07T14:19:36.083574+0000","172.21.15.142:6800/1202860895":"2024-09-07T14:19:36.083574+0000","172.21.15.142:0/382683652":"2024-09-07T14:19:36.083574+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-06T14:24:48.460 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:48.463+0000 7f4c62410640 1 -- 172.21.15.142:0/3613057369 >> v2:172.21.15.142:6800/853729507 conn(0x7f4c480775c0 msgr2=0x7f4c48079a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:48.461 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:48.464+0000 7f4c62410640 1 --2- 172.21.15.142:0/3613057369 >> v2:172.21.15.142:6800/853729507 conn(0x7f4c480775c0 0x7f4c48079a80 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c009990 tx=0x7f4c4c009110 comp rx=0 tx=0).stop 2024-09-06T14:24:48.461 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:48.464+0000 7f4c62410640 1 -- 172.21.15.142:0/3613057369 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4c5c10f110 msgr2=0x7f4c5c1a0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:48.461 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:48.464+0000 7f4c62410640 1 --2- 172.21.15.142:0/3613057369 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4c5c10f110 0x7f4c5c1a0200 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4c5402abe0 tx=0x7f4c540305d0 comp rx=0 tx=0).stop 2024-09-06T14:24:48.461 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:48.464+0000 7f4c62410640 1 -- 172.21.15.142:0/3613057369 shutdown_connections 2024-09-06T14:24:48.461 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:48.464+0000 7f4c62410640 1 --2- 172.21.15.142:0/3613057369 >> v2:172.21.15.142:6800/853729507 conn(0x7f4c480775c0 0x7f4c48079a80 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:48.461 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:48.464+0000 7f4c62410640 1 --2- 172.21.15.142:0/3613057369 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4c5c10f110 0x7f4c5c1a0200 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:48.461 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:48.464+0000 7f4c62410640 1 --2- 172.21.15.142:0/3613057369 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4c5c075b60 0x7f4c5c19ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:48.461 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:48.464+0000 7f4c62410640 1 --2- 172.21.15.142:0/3613057369 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4c5c075240 0x7f4c5c19a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:48.461 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:48.464+0000 7f4c62410640 1 -- 172.21.15.142:0/3613057369 >> 172.21.15.142:0/3613057369 conn(0x7f4c5c0fe570 msgr2=0x7f4c5c106b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:24:48.461 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:48.464+0000 7f4c62410640 1 -- 172.21.15.142:0/3613057369 shutdown_connections 2024-09-06T14:24:48.461 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:48.464+0000 7f4c62410640 1 -- 172.21.15.142:0/3613057369 wait complete. 2024-09-06T14:24:48.547 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 06 14:24:48 smithi162 systemd[1]: Starting Ceph prometheus.a for ce32462e-6c5a-11ef-bce4-c7b262605968... 2024-09-06T14:24:48.891 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:48 smithi162 ceph-mon[33787]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-06T14:24:48.891 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:48 smithi162 ceph-mon[33787]: from='client.? 172.21.15.142:0/3613057369' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-06T14:24:48.892 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 06 14:24:48 smithi162 podman[60760]: 2024-09-06 14:24:48.546031763 +0000 UTC m=+0.016508635 image pull 1d3b7f56885b6dd623f1785be963aa9c195f86bc256ea454e8d02a7980b79c53 quay.io/prometheus/prometheus:v2.51.0 2024-09-06T14:24:48.892 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 06 14:24:48 smithi162 podman[60760]: 2024-09-06 14:24:48.656891351 +0000 UTC m=+0.127368223 container create fe7a84ccb7effa272383f6bc6b7ccb526f1c4ce50c78335a7dd93350af707e78 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-06T14:24:48.892 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 06 14:24:48 smithi162 podman[60760]: 2024-09-06 14:24:48.850581752 +0000 UTC m=+0.321058624 container init fe7a84ccb7effa272383f6bc6b7ccb526f1c4ce50c78335a7dd93350af707e78 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-06T14:24:48.893 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 06 14:24:48 smithi162 podman[60760]: 2024-09-06 14:24:48.853798451 +0000 UTC m=+0.324275324 container start fe7a84ccb7effa272383f6bc6b7ccb526f1c4ce50c78335a7dd93350af707e78 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-06T14:24:48.893 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 06 14:24:48 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-prometheus-a[60775]: ts=2024-09-06T14:24:48.873Z caller=main.go:617 level=info msg="Starting Prometheus Server" mode=server version="(version=2.51.0, branch=HEAD, revision=c05c15512acb675e3f6cd662a6727854e93fc024)" 2024-09-06T14:24:48.893 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 06 14:24:48 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-prometheus-a[60775]: ts=2024-09-06T14:24:48.873Z caller=main.go:622 level=info build_context="(go=go1.22.1, platform=linux/amd64, user=root@b5723e458358, date=20240319-10:54:45, tags=netgo,builtinassets,stringlabels)" 2024-09-06T14:24:48.893 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 06 14:24:48 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-prometheus-a[60775]: ts=2024-09-06T14:24:48.873Z caller=main.go:623 level=info host_details="(Linux 5.14.0-503.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Aug 22 17:03:23 UTC 2024 x86_64 smithi162 (none))" 2024-09-06T14:24:48.893 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 06 14:24:48 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-prometheus-a[60775]: ts=2024-09-06T14:24:48.873Z caller=main.go:624 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-09-06T14:24:48.893 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 06 14:24:48 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-prometheus-a[60775]: ts=2024-09-06T14:24:48.873Z caller=main.go:625 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-06T14:24:48.893 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 06 14:24:48 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-prometheus-a[60775]: ts=2024-09-06T14:24:48.874Z caller=web.go:568 level=info component=web msg="Start listening for connections" address=:9095 2024-09-06T14:24:48.893 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 06 14:24:48 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-prometheus-a[60775]: ts=2024-09-06T14:24:48.875Z caller=main.go:1129 level=info msg="Starting TSDB ..." 2024-09-06T14:24:48.893 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 06 14:24:48 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-prometheus-a[60775]: ts=2024-09-06T14:24:48.876Z caller=tls_config.go:313 level=info component=web msg="Listening on" address=[::]:9095 2024-09-06T14:24:48.893 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 06 14:24:48 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-prometheus-a[60775]: ts=2024-09-06T14:24:48.876Z caller=tls_config.go:316 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-09-06T14:24:48.893 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 06 14:24:48 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-prometheus-a[60775]: ts=2024-09-06T14:24:48.878Z caller=head.go:616 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-09-06T14:24:48.893 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 06 14:24:48 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-prometheus-a[60775]: ts=2024-09-06T14:24:48.878Z caller=head.go:698 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=2.01µs 2024-09-06T14:24:48.894 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 06 14:24:48 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-prometheus-a[60775]: ts=2024-09-06T14:24:48.878Z caller=head.go:706 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-09-06T14:24:48.894 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 06 14:24:48 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-prometheus-a[60775]: ts=2024-09-06T14:24:48.878Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-09-06T14:24:48.894 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 06 14:24:48 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-prometheus-a[60775]: ts=2024-09-06T14:24:48.878Z caller=head.go:815 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=32.45µs wal_replay_duration=203.116µs wbl_replay_duration=109ns total_replay_duration=258.234µs 2024-09-06T14:24:48.894 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 06 14:24:48 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-prometheus-a[60775]: ts=2024-09-06T14:24:48.880Z caller=main.go:1150 level=info fs_type=XFS_SUPER_MAGIC 2024-09-06T14:24:48.894 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 06 14:24:48 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-prometheus-a[60775]: ts=2024-09-06T14:24:48.880Z caller=main.go:1153 level=info msg="TSDB started" 2024-09-06T14:24:48.894 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 06 14:24:48 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-prometheus-a[60775]: ts=2024-09-06T14:24:48.880Z caller=main.go:1335 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-06T14:24:48.894 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 06 14:24:48 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-prometheus-a[60775]: ts=2024-09-06T14:24:48.895Z caller=main.go:1372 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=15.318478ms db_storage=744ns remote_storage=1µs web_handler=849ns query_engine=530ns scrape=393.17µs scrape_sd=105.551µs notify=878ns notify_sd=772ns rules=14.471145ms tracing=5.646µs 2024-09-06T14:24:48.894 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 06 14:24:48 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-prometheus-a[60775]: ts=2024-09-06T14:24:48.895Z caller=main.go:1114 level=info msg="Server is ready to receive web requests." 2024-09-06T14:24:48.894 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 06 14:24:48 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-prometheus-a[60775]: ts=2024-09-06T14:24:48.895Z caller=manager.go:163 level=info component="rule manager" msg="Starting rule manager..." 2024-09-06T14:24:49.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:48 smithi142 ceph-mon[29958]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-06T14:24:49.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:48 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/3613057369' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-06T14:24:49.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:48 smithi142 ceph-mon[37372]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-06T14:24:49.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:48 smithi142 ceph-mon[37372]: from='client.? 172.21.15.142:0/3613057369' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-06T14:24:49.297 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 06 14:24:48 smithi162 bash[60760]: fe7a84ccb7effa272383f6bc6b7ccb526f1c4ce50c78335a7dd93350af707e78 2024-09-06T14:24:49.297 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 06 14:24:48 smithi162 systemd[1]: Started Ceph prometheus.a for ce32462e-6c5a-11ef-bce4-c7b262605968. 2024-09-06T14:24:49.364 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-06T14:24:49.365 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph osd dump --format=json 2024-09-06T14:24:49.612 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:24:50.031 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:49 smithi142 ceph-mon[29958]: pgmap v196: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 1023 B/s wr, 34 op/s 2024-09-06T14:24:50.031 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:49 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:50.031 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:49 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:50.031 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:49 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:50.031 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:49 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "mgr module enable", "module": "prometheus"} : dispatch 2024-09-06T14:24:50.032 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:49 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:50.032 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:49 smithi142 ceph-mon[37372]: pgmap v196: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 1023 B/s wr, 34 op/s 2024-09-06T14:24:50.032 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:49 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:50.032 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:49 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:50.032 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:49 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:50.032 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:49 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "mgr module enable", "module": "prometheus"} : dispatch 2024-09-06T14:24:50.032 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:49 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:50.078 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:50.051+0000 7ff533fe4640 1 -- 172.21.15.142:0/2480562662 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff52c103860 msgr2=0x7ff52c103c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:50.078 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:50.051+0000 7ff533fe4640 1 --2- 172.21.15.142:0/2480562662 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff52c103860 0x7ff52c103c40 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7ff51c009860 tx=0x7ff51c02a340 comp rx=0 tx=0).stop 2024-09-06T14:24:50.078 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:50.051+0000 7ff533fe4640 1 -- 172.21.15.142:0/2480562662 shutdown_connections 2024-09-06T14:24:50.079 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:50.051+0000 7ff533fe4640 1 --2- 172.21.15.142:0/2480562662 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff52c10ce10 0x7ff52c10f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:50.079 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:50.051+0000 7ff533fe4640 1 --2- 172.21.15.142:0/2480562662 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff52c104260 0x7ff52c10c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:50.079 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:50.051+0000 7ff533fe4640 1 --2- 172.21.15.142:0/2480562662 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff52c103860 0x7ff52c103c40 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:50.079 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:50.051+0000 7ff533fe4640 1 -- 172.21.15.142:0/2480562662 >> 172.21.15.142:0/2480562662 conn(0x7ff52c0fd430 msgr2=0x7ff52c0ff850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:24:50.079 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:50.051+0000 7ff533fe4640 1 -- 172.21.15.142:0/2480562662 shutdown_connections 2024-09-06T14:24:50.079 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:50.051+0000 7ff533fe4640 1 -- 172.21.15.142:0/2480562662 wait complete. 2024-09-06T14:24:50.079 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:50.051+0000 7ff533fe4640 1 Processor -- start 2024-09-06T14:24:50.079 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:50.052+0000 7ff533fe4640 1 -- start start 2024-09-06T14:24:50.079 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:50.052+0000 7ff533fe4640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff52c103860 0x7ff52c1986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:50.079 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:50.052+0000 7ff533fe4640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff52c104260 0x7ff52c198bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:50.079 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:50.052+0000 7ff533fe4640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff52c10ce10 0x7ff52c19df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:50.079 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:50.052+0000 7ff533fe4640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7ff52c1121a0 con 0x7ff52c103860 2024-09-06T14:24:50.080 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:50.052+0000 7ff533fe4640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff52c112020 con 0x7ff52c104260 2024-09-06T14:24:50.080 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:50.052+0000 7ff533fe4640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7ff52c112320 con 0x7ff52c10ce10 2024-09-06T14:24:50.080 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:50.052+0000 7ff531d80640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff52c103860 0x7ff52c1986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:50.080 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:50.052+0000 7ff531d80640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff52c103860 0x7ff52c1986b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:37774/0 (socket says 172.21.15.142:37774) 2024-09-06T14:24:50.080 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:50.052+0000 7ff532581640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff52c10ce10 0x7ff52c19df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:50.080 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:50.052+0000 7ff531d80640 1 -- 172.21.15.142:0/239228546 learned_addr learned my addr 172.21.15.142:0/239228546 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:24:50.080 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:50.052+0000 7ff53157f640 1 --2- 172.21.15.142:0/239228546 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff52c104260 0x7ff52c198bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:50.080 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:50.052+0000 7ff531d80640 1 -- 172.21.15.142:0/239228546 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff52c10ce10 msgr2=0x7ff52c19df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:50.080 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:50.052+0000 7ff531d80640 1 --2- 172.21.15.142:0/239228546 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff52c10ce10 0x7ff52c19df90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:50.080 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:50.052+0000 7ff531d80640 1 -- 172.21.15.142:0/239228546 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff52c104260 msgr2=0x7ff52c198bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:50.080 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:50.052+0000 7ff531d80640 1 --2- 172.21.15.142:0/239228546 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff52c104260 0x7ff52c198bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:50.080 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:50.052+0000 7ff531d80640 1 -- 172.21.15.142:0/239228546 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff52c19e760 con 0x7ff52c103860 2024-09-06T14:24:50.080 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:50.052+0000 7ff53157f640 1 --2- 172.21.15.142:0/239228546 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff52c104260 0x7ff52c198bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-06T14:24:50.081 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:50.052+0000 7ff531d80640 1 --2- 172.21.15.142:0/239228546 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff52c103860 0x7ff52c1986b0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7ff51c009980 tx=0x7ff51c033d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:24:50.081 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:50.053+0000 7ff522ffd640 1 -- 172.21.15.142:0/239228546 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff51c03f020 con 0x7ff52c103860 2024-09-06T14:24:50.081 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:50.053+0000 7ff522ffd640 1 -- 172.21.15.142:0/239228546 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7ff51c0044c0 con 0x7ff52c103860 2024-09-06T14:24:50.081 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:50.053+0000 7ff533fe4640 1 -- 172.21.15.142:0/239228546 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff52c19e9f0 con 0x7ff52c103860 2024-09-06T14:24:50.081 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:50.053+0000 7ff522ffd640 1 -- 172.21.15.142:0/239228546 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff51c037040 con 0x7ff52c103860 2024-09-06T14:24:50.081 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:50.053+0000 7ff533fe4640 1 -- 172.21.15.142:0/239228546 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff52c1a5110 con 0x7ff52c103860 2024-09-06T14:24:50.081 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:50.054+0000 7ff533fe4640 1 -- 172.21.15.142:0/239228546 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff52c103c40 con 0x7ff52c103860 2024-09-06T14:24:50.081 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:50.054+0000 7ff522ffd640 1 -- 172.21.15.142:0/239228546 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 17) ==== 99851+0+0 (secure 0 0 0) 0x7ff51c042020 con 0x7ff52c103860 2024-09-06T14:24:50.081 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:50.055+0000 7ff522ffd640 1 --2- 172.21.15.142:0/239228546 >> v2:172.21.15.142:6800/853729507 conn(0x7ff518077610 0x7ff518079ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:50.081 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:50.055+0000 7ff53157f640 1 -- 172.21.15.142:0/239228546 >> v2:172.21.15.142:6800/853729507 conn(0x7ff518077610 msgr2=0x7ff518079ad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:6800/853729507 2024-09-06T14:24:50.081 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:50.055+0000 7ff522ffd640 1 -- 172.21.15.142:0/239228546 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 5856+0+0 (secure 0 0 0) 0x7ff51c0b98a0 con 0x7ff52c103860 2024-09-06T14:24:50.081 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:50.055+0000 7ff53157f640 1 --2- 172.21.15.142:0/239228546 >> v2:172.21.15.142:6800/853729507 conn(0x7ff518077610 0x7ff518079ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-06T14:24:50.081 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:50.059+0000 7ff522ffd640 1 -- 172.21.15.142:0/239228546 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7ff51c049020 con 0x7ff52c103860 2024-09-06T14:24:50.215 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:50.218+0000 7ff533fe4640 1 -- 172.21.15.142:0/239228546 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff52c107e60 con 0x7ff52c103860 2024-09-06T14:24:50.216 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:50.219+0000 7ff522ffd640 1 -- 172.21.15.142:0/239228546 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v61) ==== 74+0+20638 (secure 0 0 0) 0x7ff51c082a20 con 0x7ff52c103860 2024-09-06T14:24:50.217 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:24:50.217 INFO:teuthology.orchestra.run.smithi142.stdout:{"epoch":61,"fsid":"ce32462e-6c5a-11ef-bce4-c7b262605968","created":"2024-09-06T14:19:13.103278+0000","modified":"2024-09-06T14:24:33.877886+0000","last_up_change":"2024-09-06T14:24:16.577635+0000","last_in_change":"2024-09-06T14:23:56.281210+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-06T14:22:11.149139+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-09-06T14:24:23.519434+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":55,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-09-06T14:24:24.425747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-09-06T14:24:26.069368+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-09-06T14:24:28.014662+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-09-06T14:24:30.077167+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7823801-9043-4aaf-a680-046c5b16f18b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6801","nonce":1649056378}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":1649056378}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":1649056378}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6803","nonce":1649056378}]},"public_addr":"172.21.15.142:6801/1649056378","cluster_addr":"172.21.15.142:6802/1649056378","heartbeat_back_addr":"172.21.15.142:6804/1649056378","heartbeat_front_addr":"172.21.15.142:6803/1649056378","state":["exists","up"]},{"osd":1,"uuid":"364b1ffb-bdda-4b70-bd6c-3bcb73110359","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6805","nonce":599802221}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":599802221}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":599802221}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6807","nonce":599802221}]},"public_addr":"172.21.15.142:6805/599802221","cluster_addr":"172.21.15.142:6806/599802221","heartbeat_back_addr":"172.21.15.142:6808/599802221","heartbeat_front_addr":"172.21.15.142:6807/599802221","state":["exists","up"]},{"osd":2,"uuid":"d7b4c863-74aa-476b-9377-8262e2c77ae3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6809","nonce":1302270091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":1302270091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":1302270091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6811","nonce":1302270091}]},"public_addr":"172.21.15.142:6809/1302270091","cluster_addr":"172.21.15.142:6810/1302270091","heartbeat_back_addr":"172.21.15.142:6812/1302270091","heartbeat_front_addr":"172.21.15.142:6811/1302270091","state":["exists","up"]},{"osd":3,"uuid":"d8080de9-24b0-4d10-b745-ec2a23f80e49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6813","nonce":2917907718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":2917907718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":2917907718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6815","nonce":2917907718}]},"public_addr":"172.21.15.142:6813/2917907718","cluster_addr":"172.21.15.142:6814/2917907718","heartbeat_back_addr":"172.21.15.142:6816/2917907718","heartbeat_front_addr":"172.21.15.142:6815/2917907718","state":["exists","up"]},{"osd":4,"uuid":"d8c6b71c-01ef-44a5-b78b-49ecce11f357","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":2665827777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6801","nonce":2665827777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6803","nonce":2665827777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":2665827777}]},"public_addr":"172.21.15.162:6800/2665827777","cluster_addr":"172.21.15.162:6801/2665827777","heartbeat_back_addr":"172.21.15.162:6803/2665827777","heartbeat_front_addr":"172.21.15.162:6802/2665827777","state":["exists","up"]},{"osd":5,"uuid":"9215d7b7-c268-44bb-b688-74ab1576d1c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":3404139106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6805","nonce":3404139106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6807","nonce":3404139106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":3404139106}]},"public_addr":"172.21.15.162:6804/3404139106","cluster_addr":"172.21.15.162:6805/3404139106","heartbeat_back_addr":"172.21.15.162:6807/3404139106","heartbeat_front_addr":"172.21.15.162:6806/3404139106","state":["exists","up"]},{"osd":6,"uuid":"457940ad-3624-4613-98ce-817baa5f2d1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1155174053}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6809","nonce":1155174053}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6811","nonce":1155174053}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1155174053}]},"public_addr":"172.21.15.162:6808/1155174053","cluster_addr":"172.21.15.162:6809/1155174053","heartbeat_back_addr":"172.21.15.162:6811/1155174053","heartbeat_front_addr":"172.21.15.162:6810/1155174053","state":["exists","up"]},{"osd":7,"uuid":"b92b364c-849e-4971-be37-6c3f440e7cec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1194583876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6813","nonce":1194583876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6815","nonce":1194583876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1194583876}]},"public_addr":"172.21.15.162:6812/1194583876","cluster_addr":"172.21.15.162:6813/1194583876","heartbeat_back_addr":"172.21.15.162:6815/1194583876","heartbeat_front_addr":"172.21.15.162:6814/1194583876","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-06T14:21:14.998819+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-06T14:21:42.692114+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-06T14:22:09.477210+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-06T14:22:35.347799+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-06T14:22:58.792921+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-06T14:23:23.876765+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-06T14:23:49.484604+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-06T14:24:15.163203+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.142:0/516816086":"2024-09-07T14:19:58.842070+0000","172.21.15.142:6800/3892080990":"2024-09-07T14:19:58.842070+0000","172.21.15.142:0/3098992975":"2024-09-07T14:19:58.842070+0000","172.21.15.142:0/3489797750":"2024-09-07T14:19:36.083574+0000","172.21.15.142:6800/1202860895":"2024-09-07T14:19:36.083574+0000","172.21.15.142:0/382683652":"2024-09-07T14:19:36.083574+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-06T14:24:50.218 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:50.221+0000 7ff533fe4640 1 -- 172.21.15.142:0/239228546 >> v2:172.21.15.142:6800/853729507 conn(0x7ff518077610 msgr2=0x7ff518079ad0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-06T14:24:50.219 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:50.221+0000 7ff533fe4640 1 --2- 172.21.15.142:0/239228546 >> v2:172.21.15.142:6800/853729507 conn(0x7ff518077610 0x7ff518079ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:50.219 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:50.221+0000 7ff533fe4640 1 -- 172.21.15.142:0/239228546 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff52c103860 msgr2=0x7ff52c1986b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:50.219 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:50.221+0000 7ff533fe4640 1 --2- 172.21.15.142:0/239228546 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff52c103860 0x7ff52c1986b0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7ff51c009980 tx=0x7ff51c033d30 comp rx=0 tx=0).stop 2024-09-06T14:24:50.219 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:50.222+0000 7ff533fe4640 1 -- 172.21.15.142:0/239228546 shutdown_connections 2024-09-06T14:24:50.219 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:50.222+0000 7ff533fe4640 1 --2- 172.21.15.142:0/239228546 >> v2:172.21.15.142:6800/853729507 conn(0x7ff518077610 0x7ff518079ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:50.219 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:50.222+0000 7ff533fe4640 1 --2- 172.21.15.142:0/239228546 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff52c10ce10 0x7ff52c19df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:50.219 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:50.222+0000 7ff533fe4640 1 --2- 172.21.15.142:0/239228546 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff52c104260 0x7ff52c198bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:50.219 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:50.222+0000 7ff533fe4640 1 --2- 172.21.15.142:0/239228546 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff52c103860 0x7ff52c1986b0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:50.219 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:50.222+0000 7ff533fe4640 1 -- 172.21.15.142:0/239228546 >> 172.21.15.142:0/239228546 conn(0x7ff52c0fd430 msgr2=0x7ff52c0fe190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:24:50.219 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:50.222+0000 7ff533fe4640 1 -- 172.21.15.142:0/239228546 shutdown_connections 2024-09-06T14:24:50.219 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:50.222+0000 7ff533fe4640 1 -- 172.21.15.142:0/239228546 wait complete. 2024-09-06T14:24:50.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:49 smithi162 ceph-mon[33787]: pgmap v196: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 1023 B/s wr, 34 op/s 2024-09-06T14:24:50.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:49 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:50.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:49 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:50.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:49 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:50.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:49 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd={"prefix": "mgr module enable", "module": "prometheus"} : dispatch 2024-09-06T14:24:50.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:49 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' 2024-09-06T14:24:50.297 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Sep 06 14:24:50 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-x[35596]: ignoring --setuser ceph since I am not root 2024-09-06T14:24:50.298 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Sep 06 14:24:50 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-x[35596]: ignoring --setgroup ceph since I am not root 2024-09-06T14:24:50.298 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Sep 06 14:24:50 smithi162 ceph-mgr[35600]: -- 172.21.15.162:0/1800934095 <== mon.0 v2:172.21.15.142:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5600a88430e0 con 0x5600a8820800 2024-09-06T14:24:50.458 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:24:50 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: ignoring --setuser ceph since I am not root 2024-09-06T14:24:50.459 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:24:50 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: ignoring --setgroup ceph since I am not root 2024-09-06T14:24:50.948 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-09-06T14:24:50.949 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-09-06T14:24:50.950 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-09-06T14:24:50.950 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-09-06T14:24:50.950 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-09-06T14:24:50.951 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-09-06T14:24:50.951 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-09-06T14:24:50.952 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-09-06T14:24:51.208 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:51 smithi142 ceph-mon[37372]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-06T14:24:51.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:51 smithi142 ceph-mon[37372]: mgrmap e17: y(active, since 4m), standbys: x 2024-09-06T14:24:51.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:51 smithi142 ceph-mon[37372]: from='client.? 172.21.15.142:0/239228546' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-06T14:24:51.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:51 smithi142 ceph-mon[29958]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-06T14:24:51.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:51 smithi142 ceph-mon[29958]: mgrmap e17: y(active, since 4m), standbys: x 2024-09-06T14:24:51.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:51 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/239228546' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-06T14:24:51.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:51 smithi162 ceph-mon[33787]: from='mgr.14146 172.21.15.142:0/1211974415' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-06T14:24:51.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:51 smithi162 ceph-mon[33787]: mgrmap e17: y(active, since 4m), standbys: x 2024-09-06T14:24:51.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:51 smithi162 ceph-mon[33787]: from='client.? 172.21.15.142:0/239228546' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-06T14:24:51.298 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Sep 06 14:24:51 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-x[35596]: 2024-09-06T14:24:51.247+0000 7fde5509e040 -1 mgr[py] Module not found: 'volumes' 2024-09-06T14:24:51.298 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Sep 06 14:24:51 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-x[35596]: 2024-09-06T14:24:51.247+0000 7fde5509e040 -1 mgr[py] Traceback (most recent call last): 2024-09-06T14:24:51.298 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Sep 06 14:24:51 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-x[35596]: File "/usr/share/ceph/mgr/volumes/__init__.py", line 2, in 2024-09-06T14:24:51.298 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Sep 06 14:24:51 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-x[35596]: from .module import Module 2024-09-06T14:24:51.298 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Sep 06 14:24:51 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-x[35596]: File "/usr/share/ceph/mgr/volumes/module.py", line 9, in 2024-09-06T14:24:51.298 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Sep 06 14:24:51 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-x[35596]: from .fs.volume import VolumeClient 2024-09-06T14:24:51.298 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Sep 06 14:24:51 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-x[35596]: File "/usr/share/ceph/mgr/volumes/fs/volume.py", line 12, in 2024-09-06T14:24:51.298 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Sep 06 14:24:51 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-x[35596]: from ceph.fs.earmarking import CephFSVolumeEarmarking, EarmarkException # type: ignore 2024-09-06T14:24:51.298 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Sep 06 14:24:51 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-x[35596]: ModuleNotFoundError: No module named 'ceph.fs' 2024-09-06T14:24:51.298 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Sep 06 14:24:51 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-x[35596]: 2024-09-06T14:24:51.298 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Sep 06 14:24:51 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-x[35596]: 2024-09-06T14:24:51.247+0000 7fde5509e040 -1 mgr[py] Class not found in module 'volumes' 2024-09-06T14:24:51.298 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Sep 06 14:24:51 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-x[35596]: 2024-09-06T14:24:51.247+0000 7fde5509e040 -1 mgr[py] Error loading module 'volumes': (2) No such file or directory 2024-09-06T14:24:51.446 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:24:51.447 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:24:51.461 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:24:51.753 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:24:51 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: 2024-09-06T14:24:51.501+0000 7f68031f3040 -1 mgr[py] Module not found: 'volumes' 2024-09-06T14:24:51.753 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:24:51 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: 2024-09-06T14:24:51.501+0000 7f68031f3040 -1 mgr[py] Traceback (most recent call last): 2024-09-06T14:24:51.754 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:24:51 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: File "/usr/share/ceph/mgr/volumes/__init__.py", line 2, in 2024-09-06T14:24:51.754 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:24:51 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: from .module import Module 2024-09-06T14:24:51.754 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:24:51 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: File "/usr/share/ceph/mgr/volumes/module.py", line 9, in 2024-09-06T14:24:51.754 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:24:51 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: from .fs.volume import VolumeClient 2024-09-06T14:24:51.754 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:24:51 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: File "/usr/share/ceph/mgr/volumes/fs/volume.py", line 12, in 2024-09-06T14:24:51.755 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:24:51 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: from ceph.fs.earmarking import CephFSVolumeEarmarking, EarmarkException # type: ignore 2024-09-06T14:24:51.755 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:24:51 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: ModuleNotFoundError: No module named 'ceph.fs' 2024-09-06T14:24:51.755 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:24:51 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: 2024-09-06T14:24:51.755 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:24:51 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: 2024-09-06T14:24:51.502+0000 7f68031f3040 -1 mgr[py] Class not found in module 'volumes' 2024-09-06T14:24:51.755 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:24:51 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: 2024-09-06T14:24:51.502+0000 7f68031f3040 -1 mgr[py] Error loading module 'volumes': (2) No such file or directory 2024-09-06T14:24:52.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:52.174+0000 7efc9646c640 1 -- 172.21.15.142:0/1234804727 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efc90107d50 msgr2=0x7efc9010e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:52.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:52.174+0000 7efc9646c640 1 --2- 172.21.15.142:0/1234804727 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efc90107d50 0x7efc9010e550 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7efc88009be0 tx=0x7efc8802a6c0 comp rx=0 tx=0).stop 2024-09-06T14:24:52.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:52.174+0000 7efc9646c640 1 -- 172.21.15.142:0/1234804727 shutdown_connections 2024-09-06T14:24:52.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:52.174+0000 7efc9646c640 1 --2- 172.21.15.142:0/1234804727 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efc90107d50 0x7efc9010e550 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:52.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:52.174+0000 7efc9646c640 1 --2- 172.21.15.142:0/1234804727 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efc901073d0 0x7efc90107810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:52.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:52.174+0000 7efc9646c640 1 --2- 172.21.15.142:0/1234804727 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efc901019c0 0x7efc90101da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:52.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:52.174+0000 7efc9646c640 1 -- 172.21.15.142:0/1234804727 >> 172.21.15.142:0/1234804727 conn(0x7efc900fd430 msgr2=0x7efc900ff850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:24:52.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:52.174+0000 7efc9646c640 1 -- 172.21.15.142:0/1234804727 shutdown_connections 2024-09-06T14:24:52.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:52.174+0000 7efc9646c640 1 -- 172.21.15.142:0/1234804727 wait complete. 2024-09-06T14:24:52.196 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:52.175+0000 7efc9646c640 1 Processor -- start 2024-09-06T14:24:52.196 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:52.175+0000 7efc9646c640 1 -- start start 2024-09-06T14:24:52.196 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:52.175+0000 7efc9646c640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efc901019c0 0x7efc90199440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:52.196 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:52.175+0000 7efc9646c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efc901073d0 0x7efc90199980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:52.196 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:52.175+0000 7efc9646c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efc90107d50 0x7efc901a44c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:52.196 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:52.175+0000 7efc9646c640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7efc901110e0 con 0x7efc901073d0 2024-09-06T14:24:52.196 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:52.175+0000 7efc9646c640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7efc90110f60 con 0x7efc90107d50 2024-09-06T14:24:52.196 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:52.175+0000 7efc9646c640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7efc90111260 con 0x7efc901019c0 2024-09-06T14:24:52.196 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:52.175+0000 7efc8ffff640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efc901019c0 0x7efc90199440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:52.196 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:52.175+0000 7efc87fff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efc901073d0 0x7efc90199980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:52.196 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:52.175+0000 7efc8ffff640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efc901019c0 0x7efc90199440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:43592/0 (socket says 172.21.15.142:43592) 2024-09-06T14:24:52.196 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:52.175+0000 7efc8ffff640 1 -- 172.21.15.142:0/175179698 learned_addr learned my addr 172.21.15.142:0/175179698 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:24:52.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:52.175+0000 7efc94a09640 1 --2- 172.21.15.142:0/175179698 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efc90107d50 0x7efc901a44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:52.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:52.175+0000 7efc87fff640 1 -- 172.21.15.142:0/175179698 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efc901019c0 msgr2=0x7efc90199440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:52.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:52.175+0000 7efc87fff640 1 --2- 172.21.15.142:0/175179698 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efc901019c0 0x7efc90199440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:52.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:52.175+0000 7efc87fff640 1 -- 172.21.15.142:0/175179698 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efc90107d50 msgr2=0x7efc901a44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:52.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:52.175+0000 7efc87fff640 1 --2- 172.21.15.142:0/175179698 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efc90107d50 0x7efc901a44c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:52.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:52.175+0000 7efc87fff640 1 -- 172.21.15.142:0/175179698 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc901a4a00 con 0x7efc901073d0 2024-09-06T14:24:52.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:52.175+0000 7efc8ffff640 1 --2- 172.21.15.142:0/175179698 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efc901019c0 0x7efc90199440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-06T14:24:52.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:52.175+0000 7efc94a09640 1 --2- 172.21.15.142:0/175179698 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efc90107d50 0x7efc901a44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-06T14:24:52.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:52.176+0000 7efc87fff640 1 --2- 172.21.15.142:0/175179698 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efc901073d0 0x7efc90199980 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7efc7800e9f0 tx=0x7efc7800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:24:52.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:52.176+0000 7efc8dffb640 1 -- 172.21.15.142:0/175179698 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc7800cd80 con 0x7efc901073d0 2024-09-06T14:24:52.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:52.176+0000 7efc9646c640 1 -- 172.21.15.142:0/175179698 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc901a4bd0 con 0x7efc901073d0 2024-09-06T14:24:52.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:52.176+0000 7efc8dffb640 1 -- 172.21.15.142:0/175179698 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7efc780035f0 con 0x7efc901073d0 2024-09-06T14:24:52.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:52.176+0000 7efc8dffb640 1 -- 172.21.15.142:0/175179698 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc780044a0 con 0x7efc901073d0 2024-09-06T14:24:52.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:52.176+0000 7efc9646c640 1 -- 172.21.15.142:0/175179698 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc901a5100 con 0x7efc901073d0 2024-09-06T14:24:52.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:52.177+0000 7efc9646c640 1 -- 172.21.15.142:0/175179698 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7efc900635c0 con 0x7efc901073d0 2024-09-06T14:24:52.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:52.177+0000 7efc8dffb640 1 -- 172.21.15.142:0/175179698 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 17) ==== 99851+0+0 (secure 0 0 0) 0x7efc78004000 con 0x7efc901073d0 2024-09-06T14:24:52.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:52.178+0000 7efc8dffb640 1 --2- 172.21.15.142:0/175179698 >> v2:172.21.15.142:6800/853729507 conn(0x7efc7c077610 0x7efc7c079ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:52.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:52.178+0000 7efc8ffff640 1 -- 172.21.15.142:0/175179698 >> v2:172.21.15.142:6800/853729507 conn(0x7efc7c077610 msgr2=0x7efc7c079ad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:6800/853729507 2024-09-06T14:24:52.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:52.178+0000 7efc8dffb640 1 -- 172.21.15.142:0/175179698 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 5856+0+0 (secure 0 0 0) 0x7efc78099aa0 con 0x7efc901073d0 2024-09-06T14:24:52.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:52.178+0000 7efc8ffff640 1 --2- 172.21.15.142:0/175179698 >> v2:172.21.15.142:6800/853729507 conn(0x7efc7c077610 0x7efc7c079ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-06T14:24:52.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:52.178+0000 7efc8dffb640 1 --2- 172.21.15.142:0/175179698 >> v2:172.21.15.162:6800/2665827777 conn(0x7efc7c080b40 0x7efc7c082fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:52.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:52.178+0000 7efc8dffb640 1 -- 172.21.15.142:0/175179698 --> v2:172.21.15.162:6800/2665827777 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7efc7800be50 con 0x7efc7c080b40 2024-09-06T14:24:52.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:52.178+0000 7efc8dffb640 1 -- 172.21.15.142:0/175179698 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_get_version_reply(handle=1 version=61) ==== 24+0+0 (secure 0 0 0) 0x7efc7809e2a0 con 0x7efc901073d0 2024-09-06T14:24:52.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:52.178+0000 7efc94a09640 1 --2- 172.21.15.142:0/175179698 >> v2:172.21.15.162:6800/2665827777 conn(0x7efc7c080b40 0x7efc7c082fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:52.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:52.179+0000 7efc94a09640 1 --2- 172.21.15.142:0/175179698 >> v2:172.21.15.162:6800/2665827777 conn(0x7efc7c080b40 0x7efc7c082fa0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:24:52.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:52.179+0000 7efc8dffb640 1 -- 172.21.15.142:0/175179698 <== osd.4 v2:172.21.15.162:6800/2665827777 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7efc7800be50 con 0x7efc7c080b40 2024-09-06T14:24:52.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:52.193+0000 7efc9646c640 1 -- 172.21.15.142:0/175179698 --> v2:172.21.15.162:6800/2665827777 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7efc900639c0 con 0x7efc7c080b40 2024-09-06T14:24:52.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:52.193+0000 7efc8dffb640 1 -- 172.21.15.142:0/175179698 <== osd.4 v2:172.21.15.162:6800/2665827777 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7efc900639c0 con 0x7efc7c080b40 2024-09-06T14:24:52.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:52.193+0000 7efc9646c640 1 -- 172.21.15.142:0/175179698 >> v2:172.21.15.162:6800/2665827777 conn(0x7efc7c080b40 msgr2=0x7efc7c082fa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:52.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:52.193+0000 7efc9646c640 1 --2- 172.21.15.142:0/175179698 >> v2:172.21.15.162:6800/2665827777 conn(0x7efc7c080b40 0x7efc7c082fa0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:52.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:52.193+0000 7efc9646c640 1 -- 172.21.15.142:0/175179698 >> v2:172.21.15.142:6800/853729507 conn(0x7efc7c077610 msgr2=0x7efc7c079ad0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-06T14:24:52.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:52.193+0000 7efc9646c640 1 --2- 172.21.15.142:0/175179698 >> v2:172.21.15.142:6800/853729507 conn(0x7efc7c077610 0x7efc7c079ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:52.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:52.193+0000 7efc9646c640 1 -- 172.21.15.142:0/175179698 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efc901073d0 msgr2=0x7efc90199980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:52.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:52.193+0000 7efc9646c640 1 --2- 172.21.15.142:0/175179698 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efc901073d0 0x7efc90199980 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7efc7800e9f0 tx=0x7efc7800eeb0 comp rx=0 tx=0).stop 2024-09-06T14:24:52.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:52.193+0000 7efc94a09640 1 -- 172.21.15.142:0/175179698 reap_dead start 2024-09-06T14:24:52.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:52.193+0000 7efc9646c640 1 -- 172.21.15.142:0/175179698 shutdown_connections 2024-09-06T14:24:52.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:52.193+0000 7efc9646c640 1 -- 172.21.15.142:0/175179698 >> 172.21.15.142:0/175179698 conn(0x7efc900fd430 msgr2=0x7efc900fedd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:24:52.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:52.193+0000 7efc9646c640 1 -- 172.21.15.142:0/175179698 shutdown_connections 2024-09-06T14:24:52.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:52.194+0000 7efc9646c640 1 -- 172.21.15.142:0/175179698 wait complete. 2024-09-06T14:24:52.879 INFO:teuthology.orchestra.run.smithi142.stdout:128849018904 2024-09-06T14:24:52.879 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-09-06T14:24:53.153 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:53.115+0000 7f5532b46640 1 -- 172.21.15.142:0/1760885154 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f552c10ce50 msgr2=0x7f552c10f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:53.153 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:53.115+0000 7f5532b46640 1 --2- 172.21.15.142:0/1760885154 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f552c10ce50 0x7f552c10f280 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f5524009be0 tx=0x7f552402a6c0 comp rx=0 tx=0).stop 2024-09-06T14:24:53.153 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:53.115+0000 7f5532b46640 1 -- 172.21.15.142:0/1760885154 shutdown_connections 2024-09-06T14:24:53.153 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:53.115+0000 7f5532b46640 1 --2- 172.21.15.142:0/1760885154 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f552c10ce50 0x7f552c10f280 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:53.153 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:53.115+0000 7f5532b46640 1 --2- 172.21.15.142:0/1760885154 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f552c1042a0 0x7f552c10c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:53.153 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:53.115+0000 7f5532b46640 1 --2- 172.21.15.142:0/1760885154 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f552c1038a0 0x7f552c103c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:53.153 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:53.115+0000 7f5532b46640 1 -- 172.21.15.142:0/1760885154 >> 172.21.15.142:0/1760885154 conn(0x7f552c0fd430 msgr2=0x7f552c0ff850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:24:53.154 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:53.115+0000 7f5532b46640 1 -- 172.21.15.142:0/1760885154 shutdown_connections 2024-09-06T14:24:53.154 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:53.115+0000 7f5532b46640 1 -- 172.21.15.142:0/1760885154 wait complete. 2024-09-06T14:24:53.154 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:53.116+0000 7f5532b46640 1 Processor -- start 2024-09-06T14:24:53.154 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:53.116+0000 7f5532b46640 1 -- start start 2024-09-06T14:24:53.154 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:53.116+0000 7f5532b46640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f552c1038a0 0x7f552c101720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:53.154 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:53.116+0000 7f5532b46640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f552c1042a0 0x7f552c101c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:53.154 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:53.116+0000 7f5532b46640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f552c10ce50 0x7f552c1021a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:53.154 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:53.116+0000 7f5532b46640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f552c1121a0 con 0x7f552c1042a0 2024-09-06T14:24:53.154 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:53.116+0000 7f5532b46640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f552c112020 con 0x7f552c10ce50 2024-09-06T14:24:53.154 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:53.116+0000 7f5532b46640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7f552c112320 con 0x7f552c1038a0 2024-09-06T14:24:53.154 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:53.116+0000 7f55308e2640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f552c1038a0 0x7f552c101720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:53.154 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:53.116+0000 7f552bfff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f552c1042a0 0x7f552c101c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:53.154 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:53.116+0000 7f55308e2640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f552c1038a0 0x7f552c101720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:43596/0 (socket says 172.21.15.142:43596) 2024-09-06T14:24:53.155 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:53.116+0000 7f552bfff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f552c1042a0 0x7f552c101c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:37822/0 (socket says 172.21.15.142:37822) 2024-09-06T14:24:53.155 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:53.116+0000 7f55308e2640 1 -- 172.21.15.142:0/1343057878 learned_addr learned my addr 172.21.15.142:0/1343057878 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:24:53.155 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:53.116+0000 7f55310e3640 1 --2- 172.21.15.142:0/1343057878 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f552c10ce50 0x7f552c1021a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:53.155 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:53.116+0000 7f55308e2640 1 -- 172.21.15.142:0/1343057878 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f552c10ce50 msgr2=0x7f552c1021a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:53.155 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:53.116+0000 7f55308e2640 1 --2- 172.21.15.142:0/1343057878 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f552c10ce50 0x7f552c1021a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:53.155 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:53.117+0000 7f55308e2640 1 -- 172.21.15.142:0/1343057878 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f552c1042a0 msgr2=0x7f552c101c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:53.155 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:53.117+0000 7f55308e2640 1 --2- 172.21.15.142:0/1343057878 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f552c1042a0 0x7f552c101c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:53.155 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:53.117+0000 7f55308e2640 1 -- 172.21.15.142:0/1343057878 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f552c1a6a30 con 0x7f552c1038a0 2024-09-06T14:24:53.155 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:53.117+0000 7f552bfff640 1 --2- 172.21.15.142:0/1343057878 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f552c1042a0 0x7f552c101c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-06T14:24:53.156 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:53.117+0000 7f55308e2640 1 --2- 172.21.15.142:0/1343057878 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f552c1038a0 0x7f552c101720 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f552000e9c0 tx=0x7f552000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:24:53.156 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:53.117+0000 7f5529ffb640 1 -- 172.21.15.142:0/1343057878 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f552000cde0 con 0x7f552c1038a0 2024-09-06T14:24:53.156 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:53.117+0000 7f5529ffb640 1 -- 172.21.15.142:0/1343057878 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7f55200035f0 con 0x7f552c1038a0 2024-09-06T14:24:53.156 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:53.117+0000 7f5532b46640 1 -- 172.21.15.142:0/1343057878 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f552c1a6d20 con 0x7f552c1038a0 2024-09-06T14:24:53.156 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:53.117+0000 7f5529ffb640 1 -- 172.21.15.142:0/1343057878 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55200044a0 con 0x7f552c1038a0 2024-09-06T14:24:53.156 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:53.117+0000 7f5532b46640 1 -- 172.21.15.142:0/1343057878 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f552c1a7220 con 0x7f552c1038a0 2024-09-06T14:24:53.156 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:53.118+0000 7f5532b46640 1 -- 172.21.15.142:0/1343057878 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f552c063540 con 0x7f552c1038a0 2024-09-06T14:24:53.156 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:53.119+0000 7f5529ffb640 1 -- 172.21.15.142:0/1343057878 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 17) ==== 99851+0+0 (secure 0 0 0) 0x7f5520004000 con 0x7f552c1038a0 2024-09-06T14:24:53.157 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:53.119+0000 7f5529ffb640 1 --2- 172.21.15.142:0/1343057878 >> v2:172.21.15.142:6800/853729507 conn(0x7f551c077610 0x7f551c079ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:53.157 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:53.119+0000 7f552bfff640 1 -- 172.21.15.142:0/1343057878 >> v2:172.21.15.142:6800/853729507 conn(0x7f551c077610 msgr2=0x7f551c079ad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:6800/853729507 2024-09-06T14:24:53.157 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:53.119+0000 7f5529ffb640 1 -- 172.21.15.142:0/1343057878 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(61..61 src has 1..61) ==== 5856+0+0 (secure 0 0 0) 0x7f5520099b00 con 0x7f552c1038a0 2024-09-06T14:24:53.157 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:53.119+0000 7f552bfff640 1 --2- 172.21.15.142:0/1343057878 >> v2:172.21.15.142:6800/853729507 conn(0x7f551c077610 0x7f551c079ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-06T14:24:53.157 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:53.119+0000 7f5529ffb640 1 --2- 172.21.15.142:0/1343057878 >> v2:172.21.15.142:6813/2917907718 conn(0x7f551c080bc0 0x7f551c083020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:53.157 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:53.119+0000 7f5529ffb640 1 -- 172.21.15.142:0/1343057878 --> v2:172.21.15.142:6813/2917907718 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f55200042f0 con 0x7f551c080bc0 2024-09-06T14:24:53.157 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:53.119+0000 7f5529ffb640 1 -- 172.21.15.142:0/1343057878 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_get_version_reply(handle=1 version=61) ==== 24+0+0 (secure 0 0 0) 0x7f552009e2a0 con 0x7f552c1038a0 2024-09-06T14:24:53.157 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:53.119+0000 7f55310e3640 1 --2- 172.21.15.142:0/1343057878 >> v2:172.21.15.142:6813/2917907718 conn(0x7f551c080bc0 0x7f551c083020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:53.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:53.120+0000 7f55310e3640 1 --2- 172.21.15.142:0/1343057878 >> v2:172.21.15.142:6813/2917907718 conn(0x7f551c080bc0 0x7f551c083020 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:24:53.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:53.120+0000 7f5529ffb640 1 -- 172.21.15.142:0/1343057878 <== osd.3 v2:172.21.15.142:6813/2917907718 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f55200042f0 con 0x7f551c080bc0 2024-09-06T14:24:53.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:53.134+0000 7f5532b46640 1 -- 172.21.15.142:0/1343057878 --> v2:172.21.15.142:6813/2917907718 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f552c063940 con 0x7f551c080bc0 2024-09-06T14:24:53.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:53.134+0000 7f5529ffb640 1 -- 172.21.15.142:0/1343057878 <== osd.3 v2:172.21.15.142:6813/2917907718 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f552c063940 con 0x7f551c080bc0 2024-09-06T14:24:53.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:53.135+0000 7f5532b46640 1 -- 172.21.15.142:0/1343057878 >> v2:172.21.15.142:6813/2917907718 conn(0x7f551c080bc0 msgr2=0x7f551c083020 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:53.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:53.135+0000 7f5532b46640 1 --2- 172.21.15.142:0/1343057878 >> v2:172.21.15.142:6813/2917907718 conn(0x7f551c080bc0 0x7f551c083020 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:53.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:53.135+0000 7f5532b46640 1 -- 172.21.15.142:0/1343057878 >> v2:172.21.15.142:6800/853729507 conn(0x7f551c077610 msgr2=0x7f551c079ad0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-06T14:24:53.159 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:53.135+0000 7f5532b46640 1 --2- 172.21.15.142:0/1343057878 >> v2:172.21.15.142:6800/853729507 conn(0x7f551c077610 0x7f551c079ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:53.159 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:53.135+0000 7f5532b46640 1 -- 172.21.15.142:0/1343057878 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f552c1038a0 msgr2=0x7f552c101720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:53.159 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:53.135+0000 7f5532b46640 1 --2- 172.21.15.142:0/1343057878 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f552c1038a0 0x7f552c101720 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f552000e9c0 tx=0x7f552000ee80 comp rx=0 tx=0).stop 2024-09-06T14:24:53.159 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:53.135+0000 7f55310e3640 1 -- 172.21.15.142:0/1343057878 reap_dead start 2024-09-06T14:24:53.159 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:53.135+0000 7f5532b46640 1 -- 172.21.15.142:0/1343057878 shutdown_connections 2024-09-06T14:24:53.159 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:53.135+0000 7f5532b46640 1 -- 172.21.15.142:0/1343057878 >> 172.21.15.142:0/1343057878 conn(0x7f552c0fd430 msgr2=0x7f552c0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:24:53.159 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:53.135+0000 7f5532b46640 1 -- 172.21.15.142:0/1343057878 shutdown_connections 2024-09-06T14:24:53.159 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:53.135+0000 7f5532b46640 1 -- 172.21.15.142:0/1343057878 wait complete. 2024-09-06T14:24:53.900 INFO:teuthology.orchestra.run.smithi142.stdout:107374182430 2024-09-06T14:24:53.901 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-09-06T14:24:53.968 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:24:53.969 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:24:53.969 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:24:53.969 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:24:53.978 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:24:54.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:54.478+0000 7f2847acd640 1 -- 172.21.15.142:0/525552019 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f284010ce30 msgr2=0x7f284010f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:54.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:54.478+0000 7f2847acd640 1 --2- 172.21.15.142:0/525552019 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f284010ce30 0x7f284010f290 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f2838009ca0 tx=0x7f283802a8c0 comp rx=0 tx=0).stop 2024-09-06T14:24:54.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:54.478+0000 7f2847acd640 1 -- 172.21.15.142:0/525552019 shutdown_connections 2024-09-06T14:24:54.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:54.478+0000 7f2847acd640 1 --2- 172.21.15.142:0/525552019 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f284010ce30 0x7f284010f290 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:54.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:54.478+0000 7f2847acd640 1 --2- 172.21.15.142:0/525552019 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f28401042f0 0x7f284010c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:54.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:54.478+0000 7f2847acd640 1 --2- 172.21.15.142:0/525552019 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f28401039d0 0x7f2840103db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:54.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:54.478+0000 7f2847acd640 1 -- 172.21.15.142:0/525552019 >> 172.21.15.142:0/525552019 conn(0x7f28400fd430 msgr2=0x7f28400ff850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:24:54.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:54.479+0000 7f2847acd640 1 -- 172.21.15.142:0/525552019 shutdown_connections 2024-09-06T14:24:54.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:54.479+0000 7f2847acd640 1 -- 172.21.15.142:0/525552019 wait complete. 2024-09-06T14:24:54.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:54.479+0000 7f2847acd640 1 Processor -- start 2024-09-06T14:24:54.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:54.479+0000 7f2847acd640 1 -- start start 2024-09-06T14:24:54.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:54.479+0000 7f2847acd640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f28401039d0 0x7f2840101680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:54.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:54.479+0000 7f2847acd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f28401042f0 0x7f2840101bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:54.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:54.479+0000 7f2847acd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f284010ce30 0x7f2840102100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:54.520 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:54.479+0000 7f2847acd640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f28401121b0 con 0x7f28401042f0 2024-09-06T14:24:54.520 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:54.479+0000 7f2847acd640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f2840112030 con 0x7f284010ce30 2024-09-06T14:24:54.520 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:54.479+0000 7f2847acd640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7f2840112330 con 0x7f28401039d0 2024-09-06T14:24:54.520 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:54.480+0000 7f2845869640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f28401039d0 0x7f2840101680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:54.520 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:54.480+0000 7f2845068640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f28401042f0 0x7f2840101bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:54.520 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:54.480+0000 7f2845869640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f28401039d0 0x7f2840101680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:43606/0 (socket says 172.21.15.142:43606) 2024-09-06T14:24:54.520 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:54.480+0000 7f2845869640 1 -- 172.21.15.142:0/2324467321 learned_addr learned my addr 172.21.15.142:0/2324467321 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:24:54.520 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:54.480+0000 7f284606a640 1 --2- 172.21.15.142:0/2324467321 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f284010ce30 0x7f2840102100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:54.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:54.480+0000 7f2845068640 1 -- 172.21.15.142:0/2324467321 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f28401039d0 msgr2=0x7f2840101680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:54.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:54.480+0000 7f2845068640 1 --2- 172.21.15.142:0/2324467321 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f28401039d0 0x7f2840101680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:54.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:54.480+0000 7f2845068640 1 -- 172.21.15.142:0/2324467321 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f284010ce30 msgr2=0x7f2840102100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:54.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:54.480+0000 7f2845068640 1 --2- 172.21.15.142:0/2324467321 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f284010ce30 0x7f2840102100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:54.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:54.480+0000 7f2845068640 1 -- 172.21.15.142:0/2324467321 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28401a6b50 con 0x7f28401042f0 2024-09-06T14:24:54.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:54.480+0000 7f2845068640 1 --2- 172.21.15.142:0/2324467321 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f28401042f0 0x7f2840101bc0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f283c00e9c0 tx=0x7f283c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:24:54.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:54.480+0000 7f2836ffd640 1 -- 172.21.15.142:0/2324467321 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f283c00ccc0 con 0x7f28401042f0 2024-09-06T14:24:54.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:54.480+0000 7f2836ffd640 1 -- 172.21.15.142:0/2324467321 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7f283c0108b0 con 0x7f28401042f0 2024-09-06T14:24:54.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:54.480+0000 7f2847acd640 1 -- 172.21.15.142:0/2324467321 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28401a6e40 con 0x7f28401042f0 2024-09-06T14:24:54.522 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:54.480+0000 7f2836ffd640 1 -- 172.21.15.142:0/2324467321 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f283c018680 con 0x7f28401042f0 2024-09-06T14:24:54.522 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:54.480+0000 7f2847acd640 1 -- 172.21.15.142:0/2324467321 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28401a72f0 con 0x7f28401042f0 2024-09-06T14:24:54.522 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:54.481+0000 7f2847acd640 1 -- 172.21.15.142:0/2324467321 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f28400635c0 con 0x7f28401042f0 2024-09-06T14:24:54.522 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:54.482+0000 7f2836ffd640 1 -- 172.21.15.142:0/2324467321 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 17) ==== 99851+0+0 (secure 0 0 0) 0x7f283c01f020 con 0x7f28401042f0 2024-09-06T14:24:54.522 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:54.482+0000 7f2836ffd640 1 --2- 172.21.15.142:0/2324467321 >> v2:172.21.15.142:6800/853729507 conn(0x7f282c077610 0x7f282c079ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:54.522 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:54.482+0000 7f2845869640 1 -- 172.21.15.142:0/2324467321 >> v2:172.21.15.142:6800/853729507 conn(0x7f282c077610 msgr2=0x7f282c079ad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:6800/853729507 2024-09-06T14:24:54.522 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:54.482+0000 7f2836ffd640 1 -- 172.21.15.142:0/2324467321 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 5856+0+0 (secure 0 0 0) 0x7f283c0996c0 con 0x7f28401042f0 2024-09-06T14:24:54.522 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:54.483+0000 7f2845869640 1 --2- 172.21.15.142:0/2324467321 >> v2:172.21.15.142:6800/853729507 conn(0x7f282c077610 0x7f282c079ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-06T14:24:54.522 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:54.483+0000 7f2836ffd640 1 --2- 172.21.15.142:0/2324467321 >> v2:172.21.15.162:6808/1155174053 conn(0x7f282c080b40 0x7f282c082fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:54.522 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:54.483+0000 7f2836ffd640 1 -- 172.21.15.142:0/2324467321 --> v2:172.21.15.162:6808/1155174053 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f283c00be50 con 0x7f282c080b40 2024-09-06T14:24:54.522 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:54.483+0000 7f2836ffd640 1 -- 172.21.15.142:0/2324467321 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_get_version_reply(handle=1 version=61) ==== 24+0+0 (secure 0 0 0) 0x7f283c0a0050 con 0x7f28401042f0 2024-09-06T14:24:54.522 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:54.483+0000 7f284606a640 1 --2- 172.21.15.142:0/2324467321 >> v2:172.21.15.162:6808/1155174053 conn(0x7f282c080b40 0x7f282c082fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:54.522 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:54.483+0000 7f284606a640 1 --2- 172.21.15.142:0/2324467321 >> v2:172.21.15.162:6808/1155174053 conn(0x7f282c080b40 0x7f282c082fa0 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:24:54.522 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:54.484+0000 7f2836ffd640 1 -- 172.21.15.142:0/2324467321 <== osd.6 v2:172.21.15.162:6808/1155174053 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f283c00be50 con 0x7f282c080b40 2024-09-06T14:24:54.523 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:54.497+0000 7f2847acd640 1 -- 172.21.15.142:0/2324467321 --> v2:172.21.15.162:6808/1155174053 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f28400639c0 con 0x7f282c080b40 2024-09-06T14:24:54.523 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:54.498+0000 7f2836ffd640 1 -- 172.21.15.142:0/2324467321 <== osd.6 v2:172.21.15.162:6808/1155174053 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f28400639c0 con 0x7f282c080b40 2024-09-06T14:24:54.523 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:54.498+0000 7f2847acd640 1 -- 172.21.15.142:0/2324467321 >> v2:172.21.15.162:6808/1155174053 conn(0x7f282c080b40 msgr2=0x7f282c082fa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:54.523 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:54.498+0000 7f2847acd640 1 --2- 172.21.15.142:0/2324467321 >> v2:172.21.15.162:6808/1155174053 conn(0x7f282c080b40 0x7f282c082fa0 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:54.523 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:54.498+0000 7f2847acd640 1 -- 172.21.15.142:0/2324467321 >> v2:172.21.15.142:6800/853729507 conn(0x7f282c077610 msgr2=0x7f282c079ad0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-06T14:24:54.523 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:54.498+0000 7f2847acd640 1 --2- 172.21.15.142:0/2324467321 >> v2:172.21.15.142:6800/853729507 conn(0x7f282c077610 0x7f282c079ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:54.523 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:54.498+0000 7f2847acd640 1 -- 172.21.15.142:0/2324467321 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f28401042f0 msgr2=0x7f2840101bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:54.523 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:54.498+0000 7f2847acd640 1 --2- 172.21.15.142:0/2324467321 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f28401042f0 0x7f2840101bc0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f283c00e9c0 tx=0x7f283c00ee80 comp rx=0 tx=0).stop 2024-09-06T14:24:54.523 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:54.498+0000 7f284606a640 1 -- 172.21.15.142:0/2324467321 reap_dead start 2024-09-06T14:24:54.523 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:54.498+0000 7f2847acd640 1 -- 172.21.15.142:0/2324467321 shutdown_connections 2024-09-06T14:24:54.523 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:54.498+0000 7f2847acd640 1 -- 172.21.15.142:0/2324467321 >> 172.21.15.142:0/2324467321 conn(0x7f28400fd430 msgr2=0x7f28401046d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:24:54.524 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:54.498+0000 7f2847acd640 1 -- 172.21.15.142:0/2324467321 shutdown_connections 2024-09-06T14:24:54.524 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:54.498+0000 7f2847acd640 1 -- 172.21.15.142:0/2324467321 wait complete. 2024-09-06T14:24:55.215 INFO:teuthology.orchestra.run.smithi142.stdout:180388626446 2024-09-06T14:24:55.215 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-09-06T14:24:55.429 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:55 smithi142 ceph-mon[29958]: Failed to load ceph-mgr modules: volumes 2024-09-06T14:24:55.429 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:55 smithi142 ceph-mon[29958]: Standby manager daemon x restarted 2024-09-06T14:24:55.429 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:55 smithi142 ceph-mon[29958]: Standby manager daemon x started 2024-09-06T14:24:55.429 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:55 smithi142 ceph-mon[29958]: from='mgr.? 172.21.15.162:0/1225920346' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/crt"} : dispatch 2024-09-06T14:24:55.429 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:55 smithi142 ceph-mon[29958]: from='mgr.? 172.21.15.162:0/1225920346' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-09-06T14:24:55.429 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:55 smithi142 ceph-mon[29958]: from='mgr.? 172.21.15.162:0/1225920346' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/key"} : dispatch 2024-09-06T14:24:55.430 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:55 smithi142 ceph-mon[29958]: from='mgr.? 172.21.15.162:0/1225920346' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-09-06T14:24:55.430 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:55 smithi142 ceph-mon[37372]: Failed to load ceph-mgr modules: volumes 2024-09-06T14:24:55.430 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:55 smithi142 ceph-mon[37372]: Standby manager daemon x restarted 2024-09-06T14:24:55.430 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:55 smithi142 ceph-mon[37372]: Standby manager daemon x started 2024-09-06T14:24:55.430 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:55 smithi142 ceph-mon[37372]: from='mgr.? 172.21.15.162:0/1225920346' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/crt"} : dispatch 2024-09-06T14:24:55.430 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:55 smithi142 ceph-mon[37372]: from='mgr.? 172.21.15.162:0/1225920346' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-09-06T14:24:55.430 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:55 smithi142 ceph-mon[37372]: from='mgr.? 172.21.15.162:0/1225920346' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/key"} : dispatch 2024-09-06T14:24:55.430 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:55 smithi142 ceph-mon[37372]: from='mgr.? 172.21.15.162:0/1225920346' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-09-06T14:24:55.547 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:55 smithi162 ceph-mon[33787]: Failed to load ceph-mgr modules: volumes 2024-09-06T14:24:55.547 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:55 smithi162 ceph-mon[33787]: Standby manager daemon x restarted 2024-09-06T14:24:55.547 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:55 smithi162 ceph-mon[33787]: Standby manager daemon x started 2024-09-06T14:24:55.547 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:55 smithi162 ceph-mon[33787]: from='mgr.? 172.21.15.162:0/1225920346' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/crt"} : dispatch 2024-09-06T14:24:55.547 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:55 smithi162 ceph-mon[33787]: from='mgr.? 172.21.15.162:0/1225920346' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-09-06T14:24:55.547 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:55 smithi162 ceph-mon[33787]: from='mgr.? 172.21.15.162:0/1225920346' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/key"} : dispatch 2024-09-06T14:24:55.547 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:55 smithi162 ceph-mon[33787]: from='mgr.? 172.21.15.162:0/1225920346' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-09-06T14:24:55.547 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Sep 06 14:24:55 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-x[35596]: 2024-09-06T14:24:55.052+0000 7fde5509e040 -1 log_channel(cluster) log [ERR] : Failed to load ceph-mgr modules: volumes 2024-09-06T14:24:55.547 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Sep 06 14:24:55 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-x[35596]: [06/Sep/2024:14:24:55] ENGINE Bus STARTING 2024-09-06T14:24:55.547 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Sep 06 14:24:55 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-x[35596]: CherryPy Checker: 2024-09-06T14:24:55.547 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Sep 06 14:24:55 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-x[35596]: The Application mounted at '' has an empty config. 2024-09-06T14:24:55.548 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Sep 06 14:24:55 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-x[35596]: 2024-09-06T14:24:55.548 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Sep 06 14:24:55 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-x[35596]: [06/Sep/2024:14:24:55] ENGINE Serving on http://:::9283 2024-09-06T14:24:55.548 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Sep 06 14:24:55 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-x[35596]: [06/Sep/2024:14:24:55] ENGINE Bus STARTED 2024-09-06T14:24:55.684 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:24:55 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: 2024-09-06T14:24:55.433+0000 7f68031f3040 -1 log_channel(cluster) log [ERR] : Failed to load ceph-mgr modules: volumes 2024-09-06T14:24:55.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:55.866+0000 7faac85a0640 1 -- 172.21.15.142:0/2733143045 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7faac0107d50 msgr2=0x7faac010e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:55.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:55.866+0000 7faac85a0640 1 --2- 172.21.15.142:0/2733143045 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7faac0107d50 0x7faac010e550 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7faab8009ca0 tx=0x7faab802c7c0 comp rx=0 tx=0).stop 2024-09-06T14:24:55.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:55.866+0000 7faac85a0640 1 -- 172.21.15.142:0/2733143045 shutdown_connections 2024-09-06T14:24:55.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:55.866+0000 7faac85a0640 1 --2- 172.21.15.142:0/2733143045 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7faac0107d50 0x7faac010e550 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:55.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:55.866+0000 7faac85a0640 1 --2- 172.21.15.142:0/2733143045 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faac01073d0 0x7faac0107810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:55.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:55.866+0000 7faac85a0640 1 --2- 172.21.15.142:0/2733143045 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faac01019c0 0x7faac0101da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:55.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:55.866+0000 7faac85a0640 1 -- 172.21.15.142:0/2733143045 >> 172.21.15.142:0/2733143045 conn(0x7faac00fd430 msgr2=0x7faac00ff850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:24:55.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:55.866+0000 7faac85a0640 1 -- 172.21.15.142:0/2733143045 shutdown_connections 2024-09-06T14:24:55.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:55.866+0000 7faac85a0640 1 -- 172.21.15.142:0/2733143045 wait complete. 2024-09-06T14:24:55.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:55.867+0000 7faac85a0640 1 Processor -- start 2024-09-06T14:24:55.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:55.867+0000 7faac85a0640 1 -- start start 2024-09-06T14:24:55.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:55.867+0000 7faac85a0640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faac01019c0 0x7faac0196420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:55.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:55.867+0000 7faac85a0640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7faac01073d0 0x7faac0196960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:55.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:55.867+0000 7faac85a0640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faac0107d50 0x7faac019bd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:55.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:55.867+0000 7faac85a0640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7faac0110030 con 0x7faac0107d50 2024-09-06T14:24:55.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:55.867+0000 7faac85a0640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7faac010feb0 con 0x7faac01019c0 2024-09-06T14:24:55.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:55.867+0000 7faac85a0640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7faac01101b0 con 0x7faac01073d0 2024-09-06T14:24:55.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:55.867+0000 7faac633c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faac01019c0 0x7faac0196420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:55.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:55.867+0000 7faac5b3b640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7faac01073d0 0x7faac0196960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:55.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:55.867+0000 7faac5b3b640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7faac01073d0 0x7faac0196960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:43670/0 (socket says 172.21.15.142:43670) 2024-09-06T14:24:55.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:55.867+0000 7faac5b3b640 1 -- 172.21.15.142:0/70124038 learned_addr learned my addr 172.21.15.142:0/70124038 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:24:55.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:55.868+0000 7faac6b3d640 1 --2- 172.21.15.142:0/70124038 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faac0107d50 0x7faac019bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:55.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:55.868+0000 7faac633c640 1 -- 172.21.15.142:0/70124038 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7faac01073d0 msgr2=0x7faac0196960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:55.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:55.868+0000 7faac633c640 1 --2- 172.21.15.142:0/70124038 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7faac01073d0 0x7faac0196960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:55.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:55.868+0000 7faac633c640 1 -- 172.21.15.142:0/70124038 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faac0107d50 msgr2=0x7faac019bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:55.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:55.868+0000 7faac633c640 1 --2- 172.21.15.142:0/70124038 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faac0107d50 0x7faac019bd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:55.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:55.868+0000 7faac633c640 1 -- 172.21.15.142:0/70124038 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faac019c4d0 con 0x7faac01019c0 2024-09-06T14:24:55.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:55.868+0000 7faac633c640 1 --2- 172.21.15.142:0/70124038 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faac01019c0 0x7faac0196420 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7faabc00e9f0 tx=0x7faabc00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:24:55.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:55.868+0000 7faab77fe640 1 -- 172.21.15.142:0/70124038 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faabc00ccc0 con 0x7faac01019c0 2024-09-06T14:24:55.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:55.868+0000 7faab77fe640 1 -- 172.21.15.142:0/70124038 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7faabc0108b0 con 0x7faac01019c0 2024-09-06T14:24:55.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:55.869+0000 7faac85a0640 1 -- 172.21.15.142:0/70124038 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faac019c7c0 con 0x7faac01019c0 2024-09-06T14:24:55.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:55.869+0000 7faac85a0640 1 -- 172.21.15.142:0/70124038 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faac01a3050 con 0x7faac01019c0 2024-09-06T14:24:55.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:55.869+0000 7faab77fe640 1 -- 172.21.15.142:0/70124038 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faabc0187a0 con 0x7faac01019c0 2024-09-06T14:24:55.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:55.869+0000 7faac85a0640 1 -- 172.21.15.142:0/70124038 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7faac00635c0 con 0x7faac01019c0 2024-09-06T14:24:55.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:55.870+0000 7faab77fe640 1 -- 172.21.15.142:0/70124038 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) ==== 99553+0+0 (secure 0 0 0) 0x7faabc010430 con 0x7faac01019c0 2024-09-06T14:24:55.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:55.871+0000 7faab77fe640 1 -- 172.21.15.142:0/70124038 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 6043+0+0 (secure 0 0 0) 0x7faabc0996c0 con 0x7faac01019c0 2024-09-06T14:24:55.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:55.871+0000 7faab77fe640 1 --2- 172.21.15.142:0/70124038 >> v2:172.21.15.142:6809/1302270091 conn(0x7faaac07de30 0x7faaac080270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:55.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:55.871+0000 7faab77fe640 1 -- 172.21.15.142:0/70124038 --> v2:172.21.15.142:6809/1302270091 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7faaac080900 con 0x7faaac07de30 2024-09-06T14:24:55.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:55.871+0000 7faab77fe640 1 -- 172.21.15.142:0/70124038 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7faabc0a0050 con 0x7faac01019c0 2024-09-06T14:24:55.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:55.871+0000 7faac5b3b640 1 --2- 172.21.15.142:0/70124038 >> v2:172.21.15.142:6809/1302270091 conn(0x7faaac07de30 0x7faaac080270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:55.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:55.872+0000 7faac5b3b640 1 --2- 172.21.15.142:0/70124038 >> v2:172.21.15.142:6809/1302270091 conn(0x7faaac07de30 0x7faaac080270 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:24:55.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:55.873+0000 7faab77fe640 1 -- 172.21.15.142:0/70124038 <== osd.2 v2:172.21.15.142:6809/1302270091 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7faaac080900 con 0x7faaac07de30 2024-09-06T14:24:55.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:55.886+0000 7faac85a0640 1 -- 172.21.15.142:0/70124038 --> v2:172.21.15.142:6809/1302270091 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7faac00639c0 con 0x7faaac07de30 2024-09-06T14:24:55.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:55.886+0000 7faab77fe640 1 -- 172.21.15.142:0/70124038 <== osd.2 v2:172.21.15.142:6809/1302270091 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7faac00639c0 con 0x7faaac07de30 2024-09-06T14:24:55.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:55.887+0000 7faac85a0640 1 -- 172.21.15.142:0/70124038 >> v2:172.21.15.142:6809/1302270091 conn(0x7faaac07de30 msgr2=0x7faaac080270 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:55.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:55.887+0000 7faac85a0640 1 --2- 172.21.15.142:0/70124038 >> v2:172.21.15.142:6809/1302270091 conn(0x7faaac07de30 0x7faaac080270 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:55.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:55.887+0000 7faac85a0640 1 -- 172.21.15.142:0/70124038 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faac01019c0 msgr2=0x7faac0196420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:55.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:55.887+0000 7faac85a0640 1 --2- 172.21.15.142:0/70124038 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faac01019c0 0x7faac0196420 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7faabc00e9f0 tx=0x7faabc00eeb0 comp rx=0 tx=0).stop 2024-09-06T14:24:55.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:55.887+0000 7faac85a0640 1 -- 172.21.15.142:0/70124038 shutdown_connections 2024-09-06T14:24:55.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:55.887+0000 7faac85a0640 1 --2- 172.21.15.142:0/70124038 >> v2:172.21.15.142:6809/1302270091 conn(0x7faaac07de30 0x7faaac080270 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:55.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:55.887+0000 7faac85a0640 1 --2- 172.21.15.142:0/70124038 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faac0107d50 0x7faac019bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:55.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:55.887+0000 7faac85a0640 1 --2- 172.21.15.142:0/70124038 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7faac01073d0 0x7faac0196960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:55.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:55.887+0000 7faac85a0640 1 --2- 172.21.15.142:0/70124038 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faac01019c0 0x7faac0196420 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:55.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:55.887+0000 7faac85a0640 1 -- 172.21.15.142:0/70124038 >> 172.21.15.142:0/70124038 conn(0x7faac00fd430 msgr2=0x7faac00fecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:24:55.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:55.887+0000 7faac85a0640 1 -- 172.21.15.142:0/70124038 shutdown_connections 2024-09-06T14:24:55.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:55.887+0000 7faac85a0640 1 -- 172.21.15.142:0/70124038 wait complete. 2024-09-06T14:24:55.893 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:24:56.114 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:56 smithi142 ceph-mon[37372]: mgrmap e18: y(active, since 4m), standbys: x 2024-09-06T14:24:56.114 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:56 smithi142 ceph-mon[37372]: Active manager daemon y restarted 2024-09-06T14:24:56.114 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:56 smithi142 ceph-mon[37372]: Activating manager daemon y 2024-09-06T14:24:56.115 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:56 smithi142 ceph-mon[37372]: osdmap e62: 8 total, 8 up, 8 in 2024-09-06T14:24:56.115 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:56 smithi142 ceph-mon[37372]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-06T14:24:56.115 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:56 smithi142 ceph-mon[37372]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-06T14:24:56.115 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:56 smithi142 ceph-mon[37372]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-06T14:24:56.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:56 smithi162 ceph-mon[33787]: mgrmap e18: y(active, since 4m), standbys: x 2024-09-06T14:24:56.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:56 smithi162 ceph-mon[33787]: Active manager daemon y restarted 2024-09-06T14:24:56.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:56 smithi162 ceph-mon[33787]: Activating manager daemon y 2024-09-06T14:24:56.298 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:56 smithi162 ceph-mon[33787]: osdmap e62: 8 total, 8 up, 8 in 2024-09-06T14:24:56.298 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:56 smithi162 ceph-mon[33787]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-06T14:24:56.298 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:56 smithi162 ceph-mon[33787]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-06T14:24:56.298 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:56 smithi162 ceph-mon[33787]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-06T14:24:56.298 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:56 smithi162 ceph-mon[33787]: mgrmap e19: y(active, starting, since 0.0170725s), standbys: x 2024-09-06T14:24:56.298 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:56 smithi162 ceph-mon[33787]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-06T14:24:56.298 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:56 smithi162 ceph-mon[33787]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-06T14:24:56.298 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:56 smithi162 ceph-mon[33787]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-06T14:24:56.298 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:56 smithi162 ceph-mon[33787]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-06T14:24:56.298 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:56 smithi162 ceph-mon[33787]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-06T14:24:56.298 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:56 smithi162 ceph-mon[33787]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-06T14:24:56.298 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:56 smithi162 ceph-mon[33787]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-06T14:24:56.298 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:56 smithi162 ceph-mon[33787]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-06T14:24:56.299 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:56 smithi162 ceph-mon[33787]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-06T14:24:56.299 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:56 smithi162 ceph-mon[33787]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-06T14:24:56.299 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:56 smithi162 ceph-mon[33787]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-09-06T14:24:56.299 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:56 smithi162 ceph-mon[33787]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-09-06T14:24:56.299 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:56 smithi162 ceph-mon[33787]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-09-06T14:24:56.299 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:56 smithi162 ceph-mon[33787]: Manager daemon y is now available 2024-09-06T14:24:56.299 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:56 smithi162 ceph-mon[33787]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-06T14:24:56.299 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:56 smithi162 ceph-mon[33787]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:24:56.299 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:56 smithi162 ceph-mon[33787]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-06T14:24:56.299 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:56 smithi162 ceph-mon[33787]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:24:56.299 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:56 smithi162 ceph-mon[33787]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-06T14:24:56.299 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:56 smithi162 ceph-mon[33787]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-06T14:24:56.299 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:56 smithi162 ceph-mon[33787]: from='mgr.14601 ' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-06T14:24:56.299 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:56 smithi162 ceph-mon[33787]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-06T14:24:56.300 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:56 smithi162 ceph-mon[33787]: from='mgr.14601 ' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-06T14:24:56.375 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:56 smithi142 ceph-mon[29958]: mgrmap e18: y(active, since 4m), standbys: x 2024-09-06T14:24:56.375 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:56 smithi142 ceph-mon[29958]: Active manager daemon y restarted 2024-09-06T14:24:56.376 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:56 smithi142 ceph-mon[29958]: Activating manager daemon y 2024-09-06T14:24:56.376 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:56 smithi142 ceph-mon[29958]: osdmap e62: 8 total, 8 up, 8 in 2024-09-06T14:24:56.376 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:56 smithi142 ceph-mon[29958]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-06T14:24:56.376 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:56 smithi142 ceph-mon[29958]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-06T14:24:56.376 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:56 smithi142 ceph-mon[29958]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-06T14:24:56.376 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:56 smithi142 ceph-mon[29958]: mgrmap e19: y(active, starting, since 0.0170725s), standbys: x 2024-09-06T14:24:56.376 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:56 smithi142 ceph-mon[29958]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-06T14:24:56.376 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:56 smithi142 ceph-mon[29958]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-06T14:24:56.376 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:56 smithi142 ceph-mon[29958]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-06T14:24:56.376 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:56 smithi142 ceph-mon[29958]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-06T14:24:56.376 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:56 smithi142 ceph-mon[29958]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-06T14:24:56.376 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:56 smithi142 ceph-mon[29958]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-06T14:24:56.376 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:56 smithi142 ceph-mon[29958]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-06T14:24:56.376 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:56 smithi142 ceph-mon[29958]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-06T14:24:56.377 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:56 smithi142 ceph-mon[29958]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-06T14:24:56.377 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:56 smithi142 ceph-mon[29958]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-06T14:24:56.377 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:56 smithi142 ceph-mon[29958]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-09-06T14:24:56.377 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:56 smithi142 ceph-mon[29958]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-09-06T14:24:56.377 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:56 smithi142 ceph-mon[29958]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-09-06T14:24:56.377 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:56 smithi142 ceph-mon[29958]: Manager daemon y is now available 2024-09-06T14:24:56.377 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:56 smithi142 ceph-mon[29958]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-06T14:24:56.377 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:56 smithi142 ceph-mon[29958]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:24:56.377 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:56 smithi142 ceph-mon[29958]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-06T14:24:56.377 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:56 smithi142 ceph-mon[29958]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:24:56.377 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:56 smithi142 ceph-mon[29958]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-06T14:24:56.377 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:56 smithi142 ceph-mon[29958]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-06T14:24:56.377 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:56 smithi142 ceph-mon[29958]: from='mgr.14601 ' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-06T14:24:56.377 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:56 smithi142 ceph-mon[29958]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-06T14:24:56.378 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:56 smithi142 ceph-mon[29958]: from='mgr.14601 ' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-06T14:24:56.378 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:56 smithi142 ceph-mon[37372]: mgrmap e19: y(active, starting, since 0.0170725s), standbys: x 2024-09-06T14:24:56.378 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:56 smithi142 ceph-mon[37372]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-06T14:24:56.378 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:56 smithi142 ceph-mon[37372]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-06T14:24:56.378 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:56 smithi142 ceph-mon[37372]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-06T14:24:56.378 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:56 smithi142 ceph-mon[37372]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-06T14:24:56.378 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:56 smithi142 ceph-mon[37372]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-06T14:24:56.378 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:56 smithi142 ceph-mon[37372]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-06T14:24:56.378 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:56 smithi142 ceph-mon[37372]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-06T14:24:56.378 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:56 smithi142 ceph-mon[37372]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-06T14:24:56.378 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:56 smithi142 ceph-mon[37372]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-06T14:24:56.379 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:56 smithi142 ceph-mon[37372]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-06T14:24:56.379 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:56 smithi142 ceph-mon[37372]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-09-06T14:24:56.379 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:56 smithi142 ceph-mon[37372]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-09-06T14:24:56.379 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:56 smithi142 ceph-mon[37372]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-09-06T14:24:56.379 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:56 smithi142 ceph-mon[37372]: Manager daemon y is now available 2024-09-06T14:24:56.379 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:56 smithi142 ceph-mon[37372]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-06T14:24:56.379 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:56 smithi142 ceph-mon[37372]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:24:56.379 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:56 smithi142 ceph-mon[37372]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-06T14:24:56.379 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:56 smithi142 ceph-mon[37372]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:24:56.379 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:56 smithi142 ceph-mon[37372]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-06T14:24:56.379 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:56 smithi142 ceph-mon[37372]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-06T14:24:56.379 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:56 smithi142 ceph-mon[37372]: from='mgr.14601 ' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-06T14:24:56.379 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:56 smithi142 ceph-mon[37372]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-06T14:24:56.379 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:56 smithi142 ceph-mon[37372]: from='mgr.14601 ' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-06T14:24:56.625 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:56.609+0000 7fc9b7310640 1 -- 172.21.15.142:0/3715246083 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc9b0107d70 msgr2=0x7fc9b010e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:56.625 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:56.609+0000 7fc9b7310640 1 --2- 172.21.15.142:0/3715246083 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc9b0107d70 0x7fc9b010e570 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc9a8009ca0 tx=0x7fc9a802a8c0 comp rx=0 tx=0).stop 2024-09-06T14:24:56.625 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:56.609+0000 7fc9b7310640 1 -- 172.21.15.142:0/3715246083 shutdown_connections 2024-09-06T14:24:56.625 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:56.609+0000 7fc9b7310640 1 --2- 172.21.15.142:0/3715246083 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc9b0107d70 0x7fc9b010e570 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:56.626 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:56.609+0000 7fc9b7310640 1 --2- 172.21.15.142:0/3715246083 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc9b01073f0 0x7fc9b0107830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:56.626 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:56.609+0000 7fc9b7310640 1 --2- 172.21.15.142:0/3715246083 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc9b01019e0 0x7fc9b0101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:56.626 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:56.609+0000 7fc9b7310640 1 -- 172.21.15.142:0/3715246083 >> 172.21.15.142:0/3715246083 conn(0x7fc9b00fd450 msgr2=0x7fc9b00ff870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:24:56.626 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:56.609+0000 7fc9b7310640 1 -- 172.21.15.142:0/3715246083 shutdown_connections 2024-09-06T14:24:56.626 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:56.609+0000 7fc9b7310640 1 -- 172.21.15.142:0/3715246083 wait complete. 2024-09-06T14:24:56.626 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:56.610+0000 7fc9b7310640 1 Processor -- start 2024-09-06T14:24:56.626 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:56.610+0000 7fc9b7310640 1 -- start start 2024-09-06T14:24:56.626 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:56.610+0000 7fc9b7310640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc9b01019e0 0x7fc9b0198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:56.626 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:56.610+0000 7fc9b7310640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc9b01073f0 0x7fc9b0198b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:56.626 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:56.610+0000 7fc9b7310640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc9b0107d70 0x7fc9b019df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:56.626 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:56.610+0000 7fc9b7310640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fc9b0112240 con 0x7fc9b01073f0 2024-09-06T14:24:56.626 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:56.610+0000 7fc9b7310640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc9b01120c0 con 0x7fc9b0107d70 2024-09-06T14:24:56.627 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:56.610+0000 7fc9b7310640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7fc9b01123c0 con 0x7fc9b01019e0 2024-09-06T14:24:56.627 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:56.610+0000 7fc9b48ab640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc9b01073f0 0x7fc9b0198b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:56.627 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:56.610+0000 7fc9b50ac640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc9b01019e0 0x7fc9b0198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:56.627 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:56.610+0000 7fc9b48ab640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc9b01073f0 0x7fc9b0198b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:37916/0 (socket says 172.21.15.142:37916) 2024-09-06T14:24:56.627 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:56.610+0000 7fc9b48ab640 1 -- 172.21.15.142:0/1147202055 learned_addr learned my addr 172.21.15.142:0/1147202055 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:24:56.627 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:56.611+0000 7fc9b58ad640 1 --2- 172.21.15.142:0/1147202055 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc9b0107d70 0x7fc9b019df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:56.627 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:56.611+0000 7fc9b48ab640 1 -- 172.21.15.142:0/1147202055 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc9b01019e0 msgr2=0x7fc9b0198630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:56.627 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:56.611+0000 7fc9b48ab640 1 --2- 172.21.15.142:0/1147202055 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc9b01019e0 0x7fc9b0198630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:56.627 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:56.611+0000 7fc9b48ab640 1 -- 172.21.15.142:0/1147202055 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc9b0107d70 msgr2=0x7fc9b019df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:56.627 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:56.611+0000 7fc9b48ab640 1 --2- 172.21.15.142:0/1147202055 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc9b0107d70 0x7fc9b019df10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:56.627 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:56.611+0000 7fc9b48ab640 1 -- 172.21.15.142:0/1147202055 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9b019e6e0 con 0x7fc9b01073f0 2024-09-06T14:24:56.628 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:56.611+0000 7fc9b58ad640 1 --2- 172.21.15.142:0/1147202055 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc9b0107d70 0x7fc9b019df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-06T14:24:56.628 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:56.611+0000 7fc9b48ab640 1 --2- 172.21.15.142:0/1147202055 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc9b01073f0 0x7fc9b0198b70 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fc9a0009c20 tx=0x7fc9a000b690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:24:56.628 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:56.611+0000 7fc9a67fc640 1 -- 172.21.15.142:0/1147202055 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9a001e070 con 0x7fc9b01073f0 2024-09-06T14:24:56.628 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:56.611+0000 7fc9b7310640 1 -- 172.21.15.142:0/1147202055 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9b019e9d0 con 0x7fc9b01073f0 2024-09-06T14:24:56.628 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:56.611+0000 7fc9a67fc640 1 -- 172.21.15.142:0/1147202055 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7fc9a0019890 con 0x7fc9b01073f0 2024-09-06T14:24:56.628 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:56.611+0000 7fc9a67fc640 1 -- 172.21.15.142:0/1147202055 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9a0018610 con 0x7fc9b01073f0 2024-09-06T14:24:56.628 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:56.611+0000 7fc9b7310640 1 -- 172.21.15.142:0/1147202055 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9b010f130 con 0x7fc9b01073f0 2024-09-06T14:24:56.628 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:56.612+0000 7fc9b7310640 1 -- 172.21.15.142:0/1147202055 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc9b0104ce0 con 0x7fc9b01073f0 2024-09-06T14:24:56.628 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:56.613+0000 7fc9a67fc640 1 -- 172.21.15.142:0/1147202055 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 20) ==== 99645+0+0 (secure 0 0 0) 0x7fc9a0021070 con 0x7fc9b01073f0 2024-09-06T14:24:56.628 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:56.613+0000 7fc9a67fc640 1 --2- 172.21.15.142:0/1147202055 >> v2:172.21.15.142:6800/2944545612 conn(0x7fc99c0773b0 0x7fc99c079870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:56.628 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:56.613+0000 7fc9b50ac640 1 --2- 172.21.15.142:0/1147202055 >> v2:172.21.15.142:6800/2944545612 conn(0x7fc99c0773b0 0x7fc99c079870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:56.628 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:56.613+0000 7fc9a67fc640 1 -- 172.21.15.142:0/1147202055 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 6043+0+0 (secure 0 0 0) 0x7fc9a009cd90 con 0x7fc9b01073f0 2024-09-06T14:24:56.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:56.613+0000 7fc9a67fc640 1 --2- 172.21.15.142:0/1147202055 >> v2:172.21.15.162:6812/1194583876 conn(0x7fc99c080a70 0x7fc99c082eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:56.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:56.613+0000 7fc9a67fc640 1 -- 172.21.15.142:0/1147202055 --> v2:172.21.15.162:6812/1194583876 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc9a001ae50 con 0x7fc99c080a70 2024-09-06T14:24:56.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:56.613+0000 7fc9a67fc640 1 -- 172.21.15.142:0/1147202055 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7fc9a009f480 con 0x7fc9b01073f0 2024-09-06T14:24:56.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:56.614+0000 7fc9b50ac640 1 --2- 172.21.15.142:0/1147202055 >> v2:172.21.15.142:6800/2944545612 conn(0x7fc99c0773b0 0x7fc99c079870 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fc9ac005d00 tx=0x7fc9ac00b410 comp rx=0 tx=0).ready entity=mgr.14601 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:24:56.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:56.614+0000 7fc9b58ad640 1 --2- 172.21.15.142:0/1147202055 >> v2:172.21.15.162:6812/1194583876 conn(0x7fc99c080a70 0x7fc99c082eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:56.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:56.614+0000 7fc9b58ad640 1 --2- 172.21.15.142:0/1147202055 >> v2:172.21.15.162:6812/1194583876 conn(0x7fc99c080a70 0x7fc99c082eb0 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:24:56.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:56.615+0000 7fc9a67fc640 1 -- 172.21.15.142:0/1147202055 <== osd.7 v2:172.21.15.162:6812/1194583876 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc9a001ae50 con 0x7fc99c080a70 2024-09-06T14:24:56.629 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:24:56 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: [06/Sep/2024:14:24:56] ENGINE Bus STARTING 2024-09-06T14:24:56.630 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:24:56 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: CherryPy Checker: 2024-09-06T14:24:56.630 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:24:56 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: The Application mounted at '' has an empty config. 2024-09-06T14:24:56.630 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:24:56 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: 2024-09-06T14:24:56.630 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:24:56 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: [06/Sep/2024:14:24:56] ENGINE Serving on http://:::9283 2024-09-06T14:24:56.630 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:24:56 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: [06/Sep/2024:14:24:56] ENGINE Bus STARTED 2024-09-06T14:24:56.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:56.640+0000 7fc9b7310640 1 -- 172.21.15.142:0/1147202055 --> v2:172.21.15.162:6812/1194583876 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fc9b0102c90 con 0x7fc99c080a70 2024-09-06T14:24:56.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:56.640+0000 7fc9a67fc640 1 -- 172.21.15.142:0/1147202055 <== osd.7 v2:172.21.15.162:6812/1194583876 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fc9b0102c90 con 0x7fc99c080a70 2024-09-06T14:24:56.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:56.641+0000 7fc9b7310640 1 -- 172.21.15.142:0/1147202055 >> v2:172.21.15.162:6812/1194583876 conn(0x7fc99c080a70 msgr2=0x7fc99c082eb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:56.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:56.641+0000 7fc9b7310640 1 --2- 172.21.15.142:0/1147202055 >> v2:172.21.15.162:6812/1194583876 conn(0x7fc99c080a70 0x7fc99c082eb0 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:56.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:56.641+0000 7fc9b7310640 1 -- 172.21.15.142:0/1147202055 >> v2:172.21.15.142:6800/2944545612 conn(0x7fc99c0773b0 msgr2=0x7fc99c079870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:56.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:56.641+0000 7fc9b7310640 1 --2- 172.21.15.142:0/1147202055 >> v2:172.21.15.142:6800/2944545612 conn(0x7fc99c0773b0 0x7fc99c079870 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fc9ac005d00 tx=0x7fc9ac00b410 comp rx=0 tx=0).stop 2024-09-06T14:24:56.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:56.641+0000 7fc9b7310640 1 -- 172.21.15.142:0/1147202055 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc9b01073f0 msgr2=0x7fc9b0198b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:56.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:56.641+0000 7fc9b7310640 1 --2- 172.21.15.142:0/1147202055 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc9b01073f0 0x7fc9b0198b70 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fc9a0009c20 tx=0x7fc9a000b690 comp rx=0 tx=0).stop 2024-09-06T14:24:56.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:56.641+0000 7fc9b58ad640 1 -- 172.21.15.142:0/1147202055 reap_dead start 2024-09-06T14:24:56.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:56.641+0000 7fc9b7310640 1 -- 172.21.15.142:0/1147202055 shutdown_connections 2024-09-06T14:24:56.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:56.641+0000 7fc9b7310640 1 -- 172.21.15.142:0/1147202055 >> 172.21.15.142:0/1147202055 conn(0x7fc9b00fd450 msgr2=0x7fc9b00ff840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:24:56.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:56.641+0000 7fc9b7310640 1 -- 172.21.15.142:0/1147202055 shutdown_connections 2024-09-06T14:24:56.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:56.641+0000 7fc9b7310640 1 -- 172.21.15.142:0/1147202055 wait complete. 2024-09-06T14:24:56.888 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:24:56 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: 2024-09-06T14:24:56.826+0000 7f676a0fd640 -1 log_channel(cephadm) log [ERR] : [06/Sep/2024:14:24:56] ENGINE Error in HTTPServer.serve 2024-09-06T14:24:56.888 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:24:56 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: Traceback (most recent call last): 2024-09-06T14:24:56.888 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:24:56 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-06T14:24:56.888 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:24:56 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: self._connections.run(self.expiration_interval) 2024-09-06T14:24:56.889 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:24:56 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-06T14:24:56.889 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:24:56 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: self._run(expiration_interval) 2024-09-06T14:24:56.889 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:24:56 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-06T14:24:56.889 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:24:56 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: new_conn = self._from_server_socket(self.server.socket) 2024-09-06T14:24:56.889 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:24:56 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-06T14:24:56.889 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:24:56 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-06T14:24:56.889 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:24:56 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-06T14:24:56.889 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:24:56 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: s = self.context.wrap_socket( 2024-09-06T14:24:56.889 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:24:56 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-06T14:24:56.889 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:24:56 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: return self.sslsocket_class._create( 2024-09-06T14:24:56.889 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:24:56 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-06T14:24:56.889 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:24:56 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: self.do_handshake() 2024-09-06T14:24:56.889 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:24:56 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-06T14:24:56.889 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:24:56 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: self._sslobj.do_handshake() 2024-09-06T14:24:56.890 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:24:56 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-06T14:24:56.890 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:24:56 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: 2024-09-06T14:24:57.070 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.044+0000 7f962004d640 1 -- 172.21.15.142:0/316260994 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9618108e70 msgr2=0x7f961810f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:57.070 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.044+0000 7f962004d640 1 --2- 172.21.15.142:0/316260994 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9618108e70 0x7f961810f670 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9610009be0 tx=0x7f961002a6c0 comp rx=0 tx=0).stop 2024-09-06T14:24:57.071 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.044+0000 7f962004d640 1 -- 172.21.15.142:0/316260994 shutdown_connections 2024-09-06T14:24:57.071 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.044+0000 7f962004d640 1 --2- 172.21.15.142:0/316260994 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9618108e70 0x7f961810f670 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:57.071 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.044+0000 7f962004d640 1 --2- 172.21.15.142:0/316260994 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f96181084f0 0x7f9618108930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:57.071 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.044+0000 7f962004d640 1 --2- 172.21.15.142:0/316260994 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9618102ae0 0x7f9618102ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:57.071 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.044+0000 7f962004d640 1 -- 172.21.15.142:0/316260994 >> 172.21.15.142:0/316260994 conn(0x7f96180fe530 msgr2=0x7f9618100950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:24:57.071 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.045+0000 7f962004d640 1 -- 172.21.15.142:0/316260994 shutdown_connections 2024-09-06T14:24:57.071 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.045+0000 7f962004d640 1 -- 172.21.15.142:0/316260994 wait complete. 2024-09-06T14:24:57.071 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.045+0000 7f962004d640 1 Processor -- start 2024-09-06T14:24:57.071 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.045+0000 7f962004d640 1 -- start start 2024-09-06T14:24:57.071 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.045+0000 7f962004d640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9618102ae0 0x7f9618075200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:57.071 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.045+0000 7f962004d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f96181084f0 0x7f9618075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:57.071 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.045+0000 7f962004d640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9618108e70 0x7f9618075c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:57.071 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.045+0000 7f962004d640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f9618112240 con 0x7f9618108e70 2024-09-06T14:24:57.072 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.045+0000 7f962004d640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f96181120c0 con 0x7f96181084f0 2024-09-06T14:24:57.072 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.045+0000 7f962004d640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7f96181123c0 con 0x7f9618102ae0 2024-09-06T14:24:57.072 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.045+0000 7f961dde9640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9618102ae0 0x7f9618075200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:57.072 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.045+0000 7f961e5ea640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9618108e70 0x7f9618075c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:57.072 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.046+0000 7f961e5ea640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9618108e70 0x7f9618075c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:37938/0 (socket says 172.21.15.142:37938) 2024-09-06T14:24:57.072 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.046+0000 7f961e5ea640 1 -- 172.21.15.142:0/1579387340 learned_addr learned my addr 172.21.15.142:0/1579387340 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:24:57.072 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.046+0000 7f961d5e8640 1 --2- 172.21.15.142:0/1579387340 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f96181084f0 0x7f9618075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:57.072 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.046+0000 7f961e5ea640 1 -- 172.21.15.142:0/1579387340 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9618102ae0 msgr2=0x7f9618075200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:57.072 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.046+0000 7f961e5ea640 1 --2- 172.21.15.142:0/1579387340 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9618102ae0 0x7f9618075200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:57.072 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.046+0000 7f961e5ea640 1 -- 172.21.15.142:0/1579387340 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f96181084f0 msgr2=0x7f9618075740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:57.072 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.046+0000 7f961e5ea640 1 --2- 172.21.15.142:0/1579387340 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f96181084f0 0x7f9618075740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:57.072 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.046+0000 7f961e5ea640 1 -- 172.21.15.142:0/1579387340 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96181a6ad0 con 0x7f9618108e70 2024-09-06T14:24:57.072 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.046+0000 7f961e5ea640 1 --2- 172.21.15.142:0/1579387340 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9618108e70 0x7f9618075c80 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f9610004af0 tx=0x7f9610030650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:24:57.073 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.046+0000 7f9616ffd640 1 -- 172.21.15.142:0/1579387340 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9610041020 con 0x7f9618108e70 2024-09-06T14:24:57.073 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.046+0000 7f962004d640 1 -- 172.21.15.142:0/1579387340 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96181a6d60 con 0x7f9618108e70 2024-09-06T14:24:57.073 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.046+0000 7f9616ffd640 1 -- 172.21.15.142:0/1579387340 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7f9610003970 con 0x7f9618108e70 2024-09-06T14:24:57.073 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.046+0000 7f9616ffd640 1 -- 172.21.15.142:0/1579387340 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9610036040 con 0x7f9618108e70 2024-09-06T14:24:57.073 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.046+0000 7f962004d640 1 -- 172.21.15.142:0/1579387340 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96181a72b0 con 0x7f9618108e70 2024-09-06T14:24:57.073 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.047+0000 7f962004d640 1 -- 172.21.15.142:0/1579387340 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9618063580 con 0x7f9618108e70 2024-09-06T14:24:57.073 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.048+0000 7f9616ffd640 1 -- 172.21.15.142:0/1579387340 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 20) ==== 99645+0+0 (secure 0 0 0) 0x7f9610044020 con 0x7f9618108e70 2024-09-06T14:24:57.073 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.048+0000 7f9616ffd640 1 --2- 172.21.15.142:0/1579387340 >> v2:172.21.15.142:6800/2944545612 conn(0x7f96000773b0 0x7f9600079870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:57.073 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.048+0000 7f9616ffd640 1 -- 172.21.15.142:0/1579387340 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 6043+0+0 (secure 0 0 0) 0x7f96100ba280 con 0x7f9618108e70 2024-09-06T14:24:57.073 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.048+0000 7f961dde9640 1 --2- 172.21.15.142:0/1579387340 >> v2:172.21.15.142:6800/2944545612 conn(0x7f96000773b0 0x7f9600079870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:57.074 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.048+0000 7f9616ffd640 1 --2- 172.21.15.142:0/1579387340 >> v2:172.21.15.142:6805/599802221 conn(0x7f9600080a70 0x7f9600082eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:57.074 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.048+0000 7f9616ffd640 1 -- 172.21.15.142:0/1579387340 --> v2:172.21.15.142:6805/599802221 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9610007e80 con 0x7f9600080a70 2024-09-06T14:24:57.074 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.048+0000 7f9616ffd640 1 -- 172.21.15.142:0/1579387340 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7f96100c1050 con 0x7f9618108e70 2024-09-06T14:24:57.074 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.049+0000 7f961d5e8640 1 --2- 172.21.15.142:0/1579387340 >> v2:172.21.15.142:6805/599802221 conn(0x7f9600080a70 0x7f9600082eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:57.074 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.049+0000 7f961dde9640 1 --2- 172.21.15.142:0/1579387340 >> v2:172.21.15.142:6800/2944545612 conn(0x7f96000773b0 0x7f9600079870 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f9608009530 tx=0x7f9608009260 comp rx=0 tx=0).ready entity=mgr.14601 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:24:57.074 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.049+0000 7f961d5e8640 1 --2- 172.21.15.142:0/1579387340 >> v2:172.21.15.142:6805/599802221 conn(0x7f9600080a70 0x7f9600082eb0 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:24:57.074 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.049+0000 7f9616ffd640 1 -- 172.21.15.142:0/1579387340 <== osd.1 v2:172.21.15.142:6805/599802221 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9610007e80 con 0x7f9600080a70 2024-09-06T14:24:57.074 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.063+0000 7f962004d640 1 -- 172.21.15.142:0/1579387340 --> v2:172.21.15.142:6805/599802221 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f9618063980 con 0x7f9600080a70 2024-09-06T14:24:57.074 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.063+0000 7f9616ffd640 1 -- 172.21.15.142:0/1579387340 <== osd.1 v2:172.21.15.142:6805/599802221 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f9618063980 con 0x7f9600080a70 2024-09-06T14:24:57.074 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.063+0000 7f962004d640 1 -- 172.21.15.142:0/1579387340 >> v2:172.21.15.142:6805/599802221 conn(0x7f9600080a70 msgr2=0x7f9600082eb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:57.074 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.063+0000 7f962004d640 1 --2- 172.21.15.142:0/1579387340 >> v2:172.21.15.142:6805/599802221 conn(0x7f9600080a70 0x7f9600082eb0 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:57.074 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.063+0000 7f962004d640 1 -- 172.21.15.142:0/1579387340 >> v2:172.21.15.142:6800/2944545612 conn(0x7f96000773b0 msgr2=0x7f9600079870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:57.074 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.063+0000 7f962004d640 1 --2- 172.21.15.142:0/1579387340 >> v2:172.21.15.142:6800/2944545612 conn(0x7f96000773b0 0x7f9600079870 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f9608009530 tx=0x7f9608009260 comp rx=0 tx=0).stop 2024-09-06T14:24:57.075 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.063+0000 7f962004d640 1 -- 172.21.15.142:0/1579387340 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9618108e70 msgr2=0x7f9618075c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:57.075 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.063+0000 7f962004d640 1 --2- 172.21.15.142:0/1579387340 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9618108e70 0x7f9618075c80 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f9610004af0 tx=0x7f9610030650 comp rx=0 tx=0).stop 2024-09-06T14:24:57.075 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.063+0000 7f961e5ea640 1 -- 172.21.15.142:0/1579387340 reap_dead start 2024-09-06T14:24:57.075 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.063+0000 7f962004d640 1 -- 172.21.15.142:0/1579387340 shutdown_connections 2024-09-06T14:24:57.075 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.063+0000 7f962004d640 1 -- 172.21.15.142:0/1579387340 >> 172.21.15.142:0/1579387340 conn(0x7f96180fe530 msgr2=0x7f9618100920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:24:57.075 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.063+0000 7f962004d640 1 -- 172.21.15.142:0/1579387340 shutdown_connections 2024-09-06T14:24:57.075 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.064+0000 7f962004d640 1 -- 172.21.15.142:0/1579387340 wait complete. 2024-09-06T14:24:57.297 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 06 14:24:57 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-iscsi-iscsi-a[59853]: debug there is no tcmu-runner data available 2024-09-06T14:24:57.523 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.501+0000 7f0da0d1e640 1 -- 172.21.15.142:0/2068366655 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0d9c107d50 msgr2=0x7f0d9c10e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:57.523 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.501+0000 7f0da0d1e640 1 --2- 172.21.15.142:0/2068366655 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0d9c107d50 0x7f0d9c10e550 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0d94009be0 tx=0x7f0d9402a6c0 comp rx=0 tx=0).stop 2024-09-06T14:24:57.523 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.501+0000 7f0da0d1e640 1 -- 172.21.15.142:0/2068366655 shutdown_connections 2024-09-06T14:24:57.523 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.501+0000 7f0da0d1e640 1 --2- 172.21.15.142:0/2068366655 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0d9c107d50 0x7f0d9c10e550 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:57.523 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.501+0000 7f0da0d1e640 1 --2- 172.21.15.142:0/2068366655 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0d9c1073d0 0x7f0d9c107810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:57.524 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.501+0000 7f0da0d1e640 1 --2- 172.21.15.142:0/2068366655 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0d9c1019c0 0x7f0d9c101da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:57.524 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.501+0000 7f0da0d1e640 1 -- 172.21.15.142:0/2068366655 >> 172.21.15.142:0/2068366655 conn(0x7f0d9c0fd430 msgr2=0x7f0d9c0ff850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:24:57.524 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.502+0000 7f0da0d1e640 1 -- 172.21.15.142:0/2068366655 shutdown_connections 2024-09-06T14:24:57.524 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.502+0000 7f0da0d1e640 1 -- 172.21.15.142:0/2068366655 wait complete. 2024-09-06T14:24:57.524 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.502+0000 7f0da0d1e640 1 Processor -- start 2024-09-06T14:24:57.524 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.502+0000 7f0da0d1e640 1 -- start start 2024-09-06T14:24:57.524 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.502+0000 7f0da0d1e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0d9c1019c0 0x7f0d9c198650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:57.524 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.502+0000 7f0da0d1e640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0d9c1073d0 0x7f0d9c198b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:57.524 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.502+0000 7f0da0d1e640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0d9c107d50 0x7f0d9c19df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:57.524 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.503+0000 7f0da0d1e640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f0d9c112220 con 0x7f0d9c1073d0 2024-09-06T14:24:57.524 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.503+0000 7f0da0d1e640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f0d9c1120a0 con 0x7f0d9c1019c0 2024-09-06T14:24:57.524 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.503+0000 7f0da0d1e640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7f0d9c1123a0 con 0x7f0d9c107d50 2024-09-06T14:24:57.524 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.503+0000 7f0d9ad9d640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0d9c107d50 0x7f0d9c19df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:57.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.503+0000 7f0d99d9b640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0d9c1073d0 0x7f0d9c198b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:57.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.503+0000 7f0d9ad9d640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0d9c107d50 0x7f0d9c19df30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:44918/0 (socket says 172.21.15.142:44918) 2024-09-06T14:24:57.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.503+0000 7f0d9ad9d640 1 -- 172.21.15.142:0/2604221028 learned_addr learned my addr 172.21.15.142:0/2604221028 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:24:57.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.503+0000 7f0d9a59c640 1 --2- 172.21.15.142:0/2604221028 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0d9c1019c0 0x7f0d9c198650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:57.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.503+0000 7f0d9ad9d640 1 -- 172.21.15.142:0/2604221028 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0d9c1019c0 msgr2=0x7f0d9c198650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:57.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.503+0000 7f0d9ad9d640 1 --2- 172.21.15.142:0/2604221028 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0d9c1019c0 0x7f0d9c198650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:57.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.503+0000 7f0d9ad9d640 1 -- 172.21.15.142:0/2604221028 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0d9c1073d0 msgr2=0x7f0d9c198b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:57.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.503+0000 7f0d9ad9d640 1 --2- 172.21.15.142:0/2604221028 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0d9c1073d0 0x7f0d9c198b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:57.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.503+0000 7f0d9ad9d640 1 -- 172.21.15.142:0/2604221028 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d9c19e700 con 0x7f0d9c107d50 2024-09-06T14:24:57.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.503+0000 7f0d99d9b640 1 --2- 172.21.15.142:0/2604221028 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0d9c1073d0 0x7f0d9c198b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-06T14:24:57.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.503+0000 7f0d9ad9d640 1 --2- 172.21.15.142:0/2604221028 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0d9c107d50 0x7f0d9c19df30 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0d94009b30 tx=0x7f0d940305d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:24:57.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.503+0000 7f0d8b7fe640 1 -- 172.21.15.142:0/2604221028 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d94041020 con 0x7f0d9c107d50 2024-09-06T14:24:57.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.503+0000 7f0d8b7fe640 1 -- 172.21.15.142:0/2604221028 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7f0d94003970 con 0x7f0d9c107d50 2024-09-06T14:24:57.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.503+0000 7f0da0d1e640 1 -- 172.21.15.142:0/2604221028 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d9c19e990 con 0x7f0d9c107d50 2024-09-06T14:24:57.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.503+0000 7f0d8b7fe640 1 -- 172.21.15.142:0/2604221028 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d94036040 con 0x7f0d9c107d50 2024-09-06T14:24:57.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.503+0000 7f0da0d1e640 1 -- 172.21.15.142:0/2604221028 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d9c1a50b0 con 0x7f0d9c107d50 2024-09-06T14:24:57.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.504+0000 7f0da0d1e640 1 -- 172.21.15.142:0/2604221028 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f0d9c063580 con 0x7f0d9c107d50 2024-09-06T14:24:57.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.505+0000 7f0d8b7fe640 1 -- 172.21.15.142:0/2604221028 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 21) ==== 99788+0+0 (secure 0 0 0) 0x7f0d94044020 con 0x7f0d9c107d50 2024-09-06T14:24:57.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.505+0000 7f0d8b7fe640 1 --2- 172.21.15.142:0/2604221028 >> v2:172.21.15.142:6800/2944545612 conn(0x7f0d84077590 0x7f0d84079a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:57.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.505+0000 7f0d8b7fe640 1 -- 172.21.15.142:0/2604221028 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(62..62 src has 1..62) ==== 6043+0+0 (secure 0 0 0) 0x7f0d940ba3e0 con 0x7f0d9c107d50 2024-09-06T14:24:57.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.505+0000 7f0d9a59c640 1 --2- 172.21.15.142:0/2604221028 >> v2:172.21.15.142:6800/2944545612 conn(0x7f0d84077590 0x7f0d84079a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:57.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.506+0000 7f0d8b7fe640 1 --2- 172.21.15.142:0/2604221028 >> v2:172.21.15.142:6801/1649056378 conn(0x7f0d84080c50 0x7f0d84083090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:57.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.506+0000 7f0d8b7fe640 1 -- 172.21.15.142:0/2604221028 --> v2:172.21.15.142:6801/1649056378 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0d94043e70 con 0x7f0d84080c50 2024-09-06T14:24:57.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.506+0000 7f0d8b7fe640 1 -- 172.21.15.142:0/2604221028 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7f0d940c1050 con 0x7f0d9c107d50 2024-09-06T14:24:57.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.506+0000 7f0d99d9b640 1 --2- 172.21.15.142:0/2604221028 >> v2:172.21.15.142:6801/1649056378 conn(0x7f0d84080c50 0x7f0d84083090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:57.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.506+0000 7f0d9a59c640 1 --2- 172.21.15.142:0/2604221028 >> v2:172.21.15.142:6800/2944545612 conn(0x7f0d84077590 0x7f0d84079a50 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f0d8c009530 tx=0x7f0d8c009260 comp rx=0 tx=0).ready entity=mgr.14601 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:24:57.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.506+0000 7f0d99d9b640 1 --2- 172.21.15.142:0/2604221028 >> v2:172.21.15.142:6801/1649056378 conn(0x7f0d84080c50 0x7f0d84083090 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:24:57.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.507+0000 7f0d8b7fe640 1 -- 172.21.15.142:0/2604221028 <== osd.0 v2:172.21.15.142:6801/1649056378 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0d94043e70 con 0x7f0d84080c50 2024-09-06T14:24:57.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.520+0000 7f0da0d1e640 1 -- 172.21.15.142:0/2604221028 --> v2:172.21.15.142:6801/1649056378 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f0d9c063980 con 0x7f0d84080c50 2024-09-06T14:24:57.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.520+0000 7f0d8b7fe640 1 -- 172.21.15.142:0/2604221028 <== osd.0 v2:172.21.15.142:6801/1649056378 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f0d9c063980 con 0x7f0d84080c50 2024-09-06T14:24:57.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.521+0000 7f0da0d1e640 1 -- 172.21.15.142:0/2604221028 >> v2:172.21.15.142:6801/1649056378 conn(0x7f0d84080c50 msgr2=0x7f0d84083090 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:57.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.521+0000 7f0da0d1e640 1 --2- 172.21.15.142:0/2604221028 >> v2:172.21.15.142:6801/1649056378 conn(0x7f0d84080c50 0x7f0d84083090 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:57.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.521+0000 7f0da0d1e640 1 -- 172.21.15.142:0/2604221028 >> v2:172.21.15.142:6800/2944545612 conn(0x7f0d84077590 msgr2=0x7f0d84079a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:57.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.521+0000 7f0da0d1e640 1 --2- 172.21.15.142:0/2604221028 >> v2:172.21.15.142:6800/2944545612 conn(0x7f0d84077590 0x7f0d84079a50 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f0d8c009530 tx=0x7f0d8c009260 comp rx=0 tx=0).stop 2024-09-06T14:24:57.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.521+0000 7f0da0d1e640 1 -- 172.21.15.142:0/2604221028 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0d9c107d50 msgr2=0x7f0d9c19df30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:57.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.521+0000 7f0da0d1e640 1 --2- 172.21.15.142:0/2604221028 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0d9c107d50 0x7f0d9c19df30 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0d94009b30 tx=0x7f0d940305d0 comp rx=0 tx=0).stop 2024-09-06T14:24:57.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.521+0000 7f0d9ad9d640 1 -- 172.21.15.142:0/2604221028 reap_dead start 2024-09-06T14:24:57.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.521+0000 7f0da0d1e640 1 -- 172.21.15.142:0/2604221028 shutdown_connections 2024-09-06T14:24:57.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.521+0000 7f0da0d1e640 1 -- 172.21.15.142:0/2604221028 >> 172.21.15.142:0/2604221028 conn(0x7f0d9c0fd430 msgr2=0x7f0d9c0ff820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:24:57.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.521+0000 7f0da0d1e640 1 -- 172.21.15.142:0/2604221028 shutdown_connections 2024-09-06T14:24:57.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:57.521+0000 7f0da0d1e640 1 -- 172.21.15.142:0/2604221028 wait complete. 2024-09-06T14:24:57.708 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:57 smithi142 ceph-mon[29958]: mgrmap e20: y(active, since 1.03044s), standbys: x 2024-09-06T14:24:57.709 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:57 smithi142 ceph-mon[29958]: [06/Sep/2024:14:24:56] ENGINE Bus STARTING 2024-09-06T14:24:57.709 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:57 smithi142 ceph-mon[29958]: [06/Sep/2024:14:24:56] ENGINE Serving on https://172.21.15.142:7150 2024-09-06T14:24:57.709 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:57 smithi142 ceph-mon[29958]: [06/Sep/2024:14:24:56] ENGINE Error in HTTPServer.serve 2024-09-06T14:24:57.709 INFO:journalctl@ceph.mon.a.smithi142.stdout: Traceback (most recent call last): 2024-09-06T14:24:57.709 INFO:journalctl@ceph.mon.a.smithi142.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-06T14:24:57.709 INFO:journalctl@ceph.mon.a.smithi142.stdout: self._connections.run(self.expiration_interval) 2024-09-06T14:24:57.709 INFO:journalctl@ceph.mon.a.smithi142.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-06T14:24:57.709 INFO:journalctl@ceph.mon.a.smithi142.stdout: self._run(expiration_interval) 2024-09-06T14:24:57.709 INFO:journalctl@ceph.mon.a.smithi142.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-06T14:24:57.709 INFO:journalctl@ceph.mon.a.smithi142.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-06T14:24:57.709 INFO:journalctl@ceph.mon.a.smithi142.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-06T14:24:57.709 INFO:journalctl@ceph.mon.a.smithi142.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-06T14:24:57.710 INFO:journalctl@ceph.mon.a.smithi142.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-06T14:24:57.710 INFO:journalctl@ceph.mon.a.smithi142.stdout: s = self.context.wrap_socket( 2024-09-06T14:24:57.710 INFO:journalctl@ceph.mon.a.smithi142.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-06T14:24:57.710 INFO:journalctl@ceph.mon.a.smithi142.stdout: return self.sslsocket_class._create( 2024-09-06T14:24:57.710 INFO:journalctl@ceph.mon.a.smithi142.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-06T14:24:57.710 INFO:journalctl@ceph.mon.a.smithi142.stdout: self.do_handshake() 2024-09-06T14:24:57.710 INFO:journalctl@ceph.mon.a.smithi142.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-06T14:24:57.710 INFO:journalctl@ceph.mon.a.smithi142.stdout: self._sslobj.do_handshake() 2024-09-06T14:24:57.710 INFO:journalctl@ceph.mon.a.smithi142.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-06T14:24:57.710 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:57 smithi142 ceph-mon[29958]: [06/Sep/2024:14:24:56] ENGINE Serving on http://172.21.15.142:8765 2024-09-06T14:24:57.710 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:57 smithi142 ceph-mon[29958]: [06/Sep/2024:14:24:56] ENGINE Bus STARTED 2024-09-06T14:24:57.710 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:57 smithi142 ceph-mon[29958]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-06T14:24:57.710 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:57 smithi142 ceph-mon[37372]: mgrmap e20: y(active, since 1.03044s), standbys: x 2024-09-06T14:24:57.711 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:57 smithi142 ceph-mon[37372]: [06/Sep/2024:14:24:56] ENGINE Bus STARTING 2024-09-06T14:24:57.711 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:57 smithi142 ceph-mon[37372]: [06/Sep/2024:14:24:56] ENGINE Serving on https://172.21.15.142:7150 2024-09-06T14:24:57.711 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:57 smithi142 ceph-mon[37372]: [06/Sep/2024:14:24:56] ENGINE Error in HTTPServer.serve 2024-09-06T14:24:57.711 INFO:journalctl@ceph.mon.c.smithi142.stdout: Traceback (most recent call last): 2024-09-06T14:24:57.711 INFO:journalctl@ceph.mon.c.smithi142.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-06T14:24:57.711 INFO:journalctl@ceph.mon.c.smithi142.stdout: self._connections.run(self.expiration_interval) 2024-09-06T14:24:57.711 INFO:journalctl@ceph.mon.c.smithi142.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-06T14:24:57.711 INFO:journalctl@ceph.mon.c.smithi142.stdout: self._run(expiration_interval) 2024-09-06T14:24:57.711 INFO:journalctl@ceph.mon.c.smithi142.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-06T14:24:57.711 INFO:journalctl@ceph.mon.c.smithi142.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-06T14:24:57.711 INFO:journalctl@ceph.mon.c.smithi142.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-06T14:24:57.711 INFO:journalctl@ceph.mon.c.smithi142.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-06T14:24:57.711 INFO:journalctl@ceph.mon.c.smithi142.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-06T14:24:57.711 INFO:journalctl@ceph.mon.c.smithi142.stdout: s = self.context.wrap_socket( 2024-09-06T14:24:57.712 INFO:journalctl@ceph.mon.c.smithi142.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-06T14:24:57.712 INFO:journalctl@ceph.mon.c.smithi142.stdout: return self.sslsocket_class._create( 2024-09-06T14:24:57.712 INFO:journalctl@ceph.mon.c.smithi142.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-06T14:24:57.712 INFO:journalctl@ceph.mon.c.smithi142.stdout: self.do_handshake() 2024-09-06T14:24:57.712 INFO:journalctl@ceph.mon.c.smithi142.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-06T14:24:57.712 INFO:journalctl@ceph.mon.c.smithi142.stdout: self._sslobj.do_handshake() 2024-09-06T14:24:57.712 INFO:journalctl@ceph.mon.c.smithi142.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-06T14:24:57.712 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:57 smithi142 ceph-mon[37372]: [06/Sep/2024:14:24:56] ENGINE Serving on http://172.21.15.142:8765 2024-09-06T14:24:57.712 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:57 smithi142 ceph-mon[37372]: [06/Sep/2024:14:24:56] ENGINE Bus STARTED 2024-09-06T14:24:57.712 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:57 smithi142 ceph-mon[37372]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-06T14:24:57.797 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:57 smithi162 ceph-mon[33787]: mgrmap e20: y(active, since 1.03044s), standbys: x 2024-09-06T14:24:57.797 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:57 smithi162 ceph-mon[33787]: [06/Sep/2024:14:24:56] ENGINE Bus STARTING 2024-09-06T14:24:57.797 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:57 smithi162 ceph-mon[33787]: [06/Sep/2024:14:24:56] ENGINE Serving on https://172.21.15.142:7150 2024-09-06T14:24:57.797 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:57 smithi162 ceph-mon[33787]: [06/Sep/2024:14:24:56] ENGINE Error in HTTPServer.serve 2024-09-06T14:24:57.797 INFO:journalctl@ceph.mon.b.smithi162.stdout: Traceback (most recent call last): 2024-09-06T14:24:57.797 INFO:journalctl@ceph.mon.b.smithi162.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-06T14:24:57.797 INFO:journalctl@ceph.mon.b.smithi162.stdout: self._connections.run(self.expiration_interval) 2024-09-06T14:24:57.798 INFO:journalctl@ceph.mon.b.smithi162.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-06T14:24:57.798 INFO:journalctl@ceph.mon.b.smithi162.stdout: self._run(expiration_interval) 2024-09-06T14:24:57.798 INFO:journalctl@ceph.mon.b.smithi162.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-06T14:24:57.798 INFO:journalctl@ceph.mon.b.smithi162.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-06T14:24:57.798 INFO:journalctl@ceph.mon.b.smithi162.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-06T14:24:57.798 INFO:journalctl@ceph.mon.b.smithi162.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-06T14:24:57.798 INFO:journalctl@ceph.mon.b.smithi162.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-06T14:24:57.798 INFO:journalctl@ceph.mon.b.smithi162.stdout: s = self.context.wrap_socket( 2024-09-06T14:24:57.798 INFO:journalctl@ceph.mon.b.smithi162.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-06T14:24:57.798 INFO:journalctl@ceph.mon.b.smithi162.stdout: return self.sslsocket_class._create( 2024-09-06T14:24:57.798 INFO:journalctl@ceph.mon.b.smithi162.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-06T14:24:57.798 INFO:journalctl@ceph.mon.b.smithi162.stdout: self.do_handshake() 2024-09-06T14:24:57.798 INFO:journalctl@ceph.mon.b.smithi162.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-06T14:24:57.798 INFO:journalctl@ceph.mon.b.smithi162.stdout: self._sslobj.do_handshake() 2024-09-06T14:24:57.799 INFO:journalctl@ceph.mon.b.smithi162.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-06T14:24:57.799 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:57 smithi162 ceph-mon[33787]: [06/Sep/2024:14:24:56] ENGINE Serving on http://172.21.15.142:8765 2024-09-06T14:24:57.799 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:57 smithi162 ceph-mon[33787]: [06/Sep/2024:14:24:56] ENGINE Bus STARTED 2024-09-06T14:24:57.799 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:57 smithi162 ceph-mon[33787]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-06T14:24:58.238 INFO:teuthology.orchestra.run.smithi142.stdout:34359738414 2024-09-06T14:24:58.238 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-09-06T14:24:58.547 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:58 smithi162 ceph-mon[33787]: mgrmap e21: y(active, since 2s), standbys: x 2024-09-06T14:24:58.547 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:58 smithi162 ceph-mon[33787]: pgmap v4: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail 2024-09-06T14:24:58.638 INFO:teuthology.orchestra.run.smithi142.stdout:77309411362 2024-09-06T14:24:58.639 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-09-06T14:24:58.760 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:58 smithi142 ceph-mon[29958]: mgrmap e21: y(active, since 2s), standbys: x 2024-09-06T14:24:58.761 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:58 smithi142 ceph-mon[29958]: pgmap v4: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail 2024-09-06T14:24:58.761 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:58 smithi142 ceph-mon[37372]: mgrmap e21: y(active, since 2s), standbys: x 2024-09-06T14:24:58.761 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:58 smithi142 ceph-mon[37372]: pgmap v4: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail 2024-09-06T14:24:59.422 INFO:teuthology.orchestra.run.smithi142.stdout:206158430218 2024-09-06T14:24:59.422 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-09-06T14:24:59.729 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:59.731+0000 7f1dc9da5640 1 -- 172.21.15.142:0/1610091271 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1dc4108530 msgr2=0x7f1dc4108970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:59.729 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:59.731+0000 7f1dc9da5640 1 --2- 172.21.15.142:0/1610091271 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1dc4108530 0x7f1dc4108970 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f1db8009900 tx=0x7f1db802a410 comp rx=0 tx=0).stop 2024-09-06T14:24:59.729 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:59.731+0000 7f1dc9da5640 1 -- 172.21.15.142:0/1610091271 shutdown_connections 2024-09-06T14:24:59.729 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:59.731+0000 7f1dc9da5640 1 --2- 172.21.15.142:0/1610091271 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1dc4108eb0 0x7f1dc410f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:59.729 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:59.731+0000 7f1dc9da5640 1 --2- 172.21.15.142:0/1610091271 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1dc4108530 0x7f1dc4108970 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:59.730 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:59.731+0000 7f1dc9da5640 1 --2- 172.21.15.142:0/1610091271 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1dc4102b20 0x7f1dc4102f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:59.730 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:59.731+0000 7f1dc9da5640 1 -- 172.21.15.142:0/1610091271 >> 172.21.15.142:0/1610091271 conn(0x7f1dc40fe570 msgr2=0x7f1dc4100990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:24:59.730 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:59.732+0000 7f1dc9da5640 1 -- 172.21.15.142:0/1610091271 shutdown_connections 2024-09-06T14:24:59.730 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:59.732+0000 7f1dc9da5640 1 -- 172.21.15.142:0/1610091271 wait complete. 2024-09-06T14:24:59.730 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:59.732+0000 7f1dc9da5640 1 Processor -- start 2024-09-06T14:24:59.732 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:59.733+0000 7f1dc9da5640 1 -- start start 2024-09-06T14:24:59.732 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:59.733+0000 7f1dc9da5640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1dc4102b20 0x7f1dc4198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:59.732 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:59.733+0000 7f1dc9da5640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1dc4108530 0x7f1dc4198c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:59.732 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:59.733+0000 7f1dc9da5640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1dc4108eb0 0x7f1dc419dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:59.732 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:59.733+0000 7f1dc9da5640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f1dc4112280 con 0x7f1dc4108530 2024-09-06T14:24:59.732 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:59.733+0000 7f1dc9da5640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1dc4112100 con 0x7f1dc4102b20 2024-09-06T14:24:59.732 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:59.733+0000 7f1dc9da5640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7f1dc4112400 con 0x7f1dc4108eb0 2024-09-06T14:24:59.732 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:59.733+0000 7f1dc3fff640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1dc4108eb0 0x7f1dc419dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:59.732 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:59.733+0000 7f1dc2ffd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1dc4108530 0x7f1dc4198c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:59.732 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:59.733+0000 7f1dc3fff640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1dc4108eb0 0x7f1dc419dff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:44964/0 (socket says 172.21.15.142:44964) 2024-09-06T14:24:59.732 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:59.733+0000 7f1dc3fff640 1 -- 172.21.15.142:0/2262875740 learned_addr learned my addr 172.21.15.142:0/2262875740 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:24:59.732 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:59.733+0000 7f1dc37fe640 1 --2- 172.21.15.142:0/2262875740 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1dc4102b20 0x7f1dc4198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:59.733 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:59.733+0000 7f1dc3fff640 1 -- 172.21.15.142:0/2262875740 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1dc4102b20 msgr2=0x7f1dc4198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:59.733 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:59.733+0000 7f1dc3fff640 1 --2- 172.21.15.142:0/2262875740 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1dc4102b20 0x7f1dc4198710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:59.733 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:59.734+0000 7f1dc3fff640 1 -- 172.21.15.142:0/2262875740 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1dc4108530 msgr2=0x7f1dc4198c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:59.733 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:59.734+0000 7f1dc3fff640 1 --2- 172.21.15.142:0/2262875740 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1dc4108530 0x7f1dc4198c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:59.733 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:59.734+0000 7f1dc3fff640 1 -- 172.21.15.142:0/2262875740 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1dc419e7c0 con 0x7f1dc4108eb0 2024-09-06T14:24:59.733 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:59.734+0000 7f1dc2ffd640 1 --2- 172.21.15.142:0/2262875740 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1dc4108530 0x7f1dc4198c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-06T14:24:59.733 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:59.734+0000 7f1dc3fff640 1 --2- 172.21.15.142:0/2262875740 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1dc4108eb0 0x7f1dc419dff0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f1dbc00e9b0 tx=0x7f1dbc00ee70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:24:59.733 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:59.734+0000 7f1dc0ff9640 1 -- 172.21.15.142:0/2262875740 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dbc00cd80 con 0x7f1dc4108eb0 2024-09-06T14:24:59.733 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:59.734+0000 7f1dc0ff9640 1 -- 172.21.15.142:0/2262875740 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7f1dbc0035f0 con 0x7f1dc4108eb0 2024-09-06T14:24:59.733 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:59.734+0000 7f1dc9da5640 1 -- 172.21.15.142:0/2262875740 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1dc419eab0 con 0x7f1dc4108eb0 2024-09-06T14:24:59.733 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:59.734+0000 7f1dc0ff9640 1 -- 172.21.15.142:0/2262875740 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dbc0044a0 con 0x7f1dc4108eb0 2024-09-06T14:24:59.733 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:59.734+0000 7f1dc9da5640 1 -- 172.21.15.142:0/2262875740 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1dc40750e0 con 0x7f1dc4108eb0 2024-09-06T14:24:59.733 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:59.735+0000 7f1dc9da5640 1 -- 172.21.15.142:0/2262875740 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f1dc40635c0 con 0x7f1dc4108eb0 2024-09-06T14:24:59.735 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:59.736+0000 7f1dc0ff9640 1 -- 172.21.15.142:0/2262875740 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 21) ==== 99788+0+0 (secure 0 0 0) 0x7f1dbc004020 con 0x7f1dc4108eb0 2024-09-06T14:24:59.735 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:59.736+0000 7f1dc0ff9640 1 --2- 172.21.15.142:0/2262875740 >> v2:172.21.15.142:6800/2944545612 conn(0x7f1dac077590 0x7f1dac079a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:59.735 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:59.736+0000 7f1dc0ff9640 1 -- 172.21.15.142:0/2262875740 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(62..62 src has 1..62) ==== 6043+0+0 (secure 0 0 0) 0x7f1dbc099a00 con 0x7f1dc4108eb0 2024-09-06T14:24:59.735 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:59.736+0000 7f1dc37fe640 1 --2- 172.21.15.142:0/2262875740 >> v2:172.21.15.142:6800/2944545612 conn(0x7f1dac077590 0x7f1dac079a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:59.735 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:59.736+0000 7f1dc0ff9640 1 --2- 172.21.15.142:0/2262875740 >> v2:172.21.15.162:6804/3404139106 conn(0x7f1dac080c50 0x7f1dac083090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:24:59.735 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:59.736+0000 7f1dc0ff9640 1 -- 172.21.15.142:0/2262875740 --> v2:172.21.15.162:6804/3404139106 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1dbc015e80 con 0x7f1dac080c50 2024-09-06T14:24:59.735 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:59.736+0000 7f1dc0ff9640 1 -- 172.21.15.142:0/2262875740 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7f1dbc09be60 con 0x7f1dc4108eb0 2024-09-06T14:24:59.735 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:59.737+0000 7f1dc37fe640 1 --2- 172.21.15.142:0/2262875740 >> v2:172.21.15.142:6800/2944545612 conn(0x7f1dac077590 0x7f1dac079a50 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1db4005c40 tx=0x7f1db4010040 comp rx=0 tx=0).ready entity=mgr.14601 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:24:59.735 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:59.737+0000 7f1dc2ffd640 1 --2- 172.21.15.142:0/2262875740 >> v2:172.21.15.162:6804/3404139106 conn(0x7f1dac080c50 0x7f1dac083090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:24:59.735 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:59.737+0000 7f1dc2ffd640 1 --2- 172.21.15.142:0/2262875740 >> v2:172.21.15.162:6804/3404139106 conn(0x7f1dac080c50 0x7f1dac083090 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:24:59.735 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:59.738+0000 7f1dc0ff9640 1 -- 172.21.15.142:0/2262875740 <== osd.5 v2:172.21.15.162:6804/3404139106 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1dbc015e80 con 0x7f1dac080c50 2024-09-06T14:24:59.747 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:59.751+0000 7f1dc9da5640 1 -- 172.21.15.142:0/2262875740 --> v2:172.21.15.162:6804/3404139106 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f1dc40639c0 con 0x7f1dac080c50 2024-09-06T14:24:59.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:59.751+0000 7f1dc0ff9640 1 -- 172.21.15.142:0/2262875740 <== osd.5 v2:172.21.15.162:6804/3404139106 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f1dc40639c0 con 0x7f1dac080c50 2024-09-06T14:24:59.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:59.752+0000 7f1dc9da5640 1 -- 172.21.15.142:0/2262875740 >> v2:172.21.15.162:6804/3404139106 conn(0x7f1dac080c50 msgr2=0x7f1dac083090 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:59.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:59.752+0000 7f1dc9da5640 1 --2- 172.21.15.142:0/2262875740 >> v2:172.21.15.162:6804/3404139106 conn(0x7f1dac080c50 0x7f1dac083090 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:24:59.749 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:59.752+0000 7f1dc9da5640 1 -- 172.21.15.142:0/2262875740 >> v2:172.21.15.142:6800/2944545612 conn(0x7f1dac077590 msgr2=0x7f1dac079a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:59.749 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:59.752+0000 7f1dc9da5640 1 --2- 172.21.15.142:0/2262875740 >> v2:172.21.15.142:6800/2944545612 conn(0x7f1dac077590 0x7f1dac079a50 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1db4005c40 tx=0x7f1db4010040 comp rx=0 tx=0).stop 2024-09-06T14:24:59.749 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:59.752+0000 7f1dc9da5640 1 -- 172.21.15.142:0/2262875740 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1dc4108eb0 msgr2=0x7f1dc419dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:24:59.749 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:59.752+0000 7f1dc9da5640 1 --2- 172.21.15.142:0/2262875740 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1dc4108eb0 0x7f1dc419dff0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f1dbc00e9b0 tx=0x7f1dbc00ee70 comp rx=0 tx=0).stop 2024-09-06T14:24:59.749 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:59.753+0000 7f1dc3fff640 1 -- 172.21.15.142:0/2262875740 reap_dead start 2024-09-06T14:24:59.749 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:59.753+0000 7f1dc9da5640 1 -- 172.21.15.142:0/2262875740 shutdown_connections 2024-09-06T14:24:59.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:59.753+0000 7f1dc9da5640 1 -- 172.21.15.142:0/2262875740 >> 172.21.15.142:0/2262875740 conn(0x7f1dc40fe570 msgr2=0x7f1dc4100960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:24:59.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:59.753+0000 7f1dc9da5640 1 -- 172.21.15.142:0/2262875740 shutdown_connections 2024-09-06T14:24:59.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:24:59.754+0000 7f1dc9da5640 1 -- 172.21.15.142:0/2262875740 wait complete. 2024-09-06T14:24:59.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:59 smithi142 ceph-mon[29958]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:24:59.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:59 smithi142 ceph-mon[29958]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:24:59.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:59 smithi142 ceph-mon[29958]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:24:59.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:59 smithi142 ceph-mon[29958]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:24:59.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:59 smithi142 ceph-mon[29958]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-06T14:24:59.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:59 smithi142 ceph-mon[29958]: from='mgr.14601 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-06T14:24:59.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:59 smithi142 ceph-mon[29958]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-06T14:24:59.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:59 smithi142 ceph-mon[29958]: from='mgr.14601 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-06T14:24:59.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:59 smithi142 ceph-mon[29958]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-06T14:24:59.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:59 smithi142 ceph-mon[29958]: from='mgr.14601 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-06T14:24:59.960 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:59 smithi142 ceph-mon[29958]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-06T14:24:59.960 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:59 smithi142 ceph-mon[29958]: from='mgr.14601 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-06T14:24:59.960 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:24:59 smithi142 ceph-mon[29958]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:24:59.960 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:59 smithi142 ceph-mon[37372]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:24:59.960 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:59 smithi142 ceph-mon[37372]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:24:59.960 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:59 smithi142 ceph-mon[37372]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:24:59.960 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:59 smithi142 ceph-mon[37372]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:24:59.960 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:59 smithi142 ceph-mon[37372]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-06T14:24:59.960 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:59 smithi142 ceph-mon[37372]: from='mgr.14601 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-06T14:24:59.960 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:59 smithi142 ceph-mon[37372]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-06T14:24:59.960 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:59 smithi142 ceph-mon[37372]: from='mgr.14601 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-06T14:24:59.960 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:59 smithi142 ceph-mon[37372]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-06T14:24:59.961 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:59 smithi142 ceph-mon[37372]: from='mgr.14601 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-06T14:24:59.961 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:59 smithi142 ceph-mon[37372]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-06T14:24:59.961 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:59 smithi142 ceph-mon[37372]: from='mgr.14601 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-06T14:24:59.961 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:24:59 smithi142 ceph-mon[37372]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:00.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:59 smithi162 ceph-mon[33787]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:00.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:59 smithi162 ceph-mon[33787]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:00.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:59 smithi162 ceph-mon[33787]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:00.048 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:59 smithi162 ceph-mon[33787]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:00.048 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:59 smithi162 ceph-mon[33787]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-06T14:25:00.048 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:59 smithi162 ceph-mon[33787]: from='mgr.14601 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-06T14:25:00.048 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:59 smithi162 ceph-mon[33787]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-06T14:25:00.048 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:59 smithi162 ceph-mon[33787]: from='mgr.14601 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-06T14:25:00.048 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:59 smithi162 ceph-mon[33787]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-06T14:25:00.048 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:59 smithi162 ceph-mon[33787]: from='mgr.14601 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-06T14:25:00.048 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:59 smithi162 ceph-mon[33787]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-06T14:25:00.048 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:59 smithi162 ceph-mon[33787]: from='mgr.14601 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-06T14:25:00.048 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:24:59 smithi162 ceph-mon[33787]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:00.721 INFO:teuthology.orchestra.run.smithi142.stdout:154618822676 2024-09-06T14:25:00.721 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-09-06T14:25:00.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:00.872+0000 7f3e84c18640 1 -- 172.21.15.142:0/2695511300 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3e80107d50 msgr2=0x7f3e8010e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:25:00.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:00.872+0000 7f3e84c18640 1 --2- 172.21.15.142:0/2695511300 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3e80107d50 0x7f3e8010e550 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f3e78009be0 tx=0x7f3e7802a6c0 comp rx=0 tx=0).stop 2024-09-06T14:25:00.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:00.872+0000 7f3e84c18640 1 -- 172.21.15.142:0/2695511300 shutdown_connections 2024-09-06T14:25:00.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:00.872+0000 7f3e84c18640 1 --2- 172.21.15.142:0/2695511300 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3e80107d50 0x7f3e8010e550 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:00.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:00.872+0000 7f3e84c18640 1 --2- 172.21.15.142:0/2695511300 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3e801073d0 0x7f3e80107810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:00.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:00.872+0000 7f3e84c18640 1 --2- 172.21.15.142:0/2695511300 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3e801019c0 0x7f3e80101da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:00.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:00.872+0000 7f3e84c18640 1 -- 172.21.15.142:0/2695511300 >> 172.21.15.142:0/2695511300 conn(0x7f3e800fd430 msgr2=0x7f3e800ff850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:25:00.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:00.872+0000 7f3e84c18640 1 -- 172.21.15.142:0/2695511300 shutdown_connections 2024-09-06T14:25:00.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:00.872+0000 7f3e84c18640 1 -- 172.21.15.142:0/2695511300 wait complete. 2024-09-06T14:25:00.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:00.873+0000 7f3e84c18640 1 Processor -- start 2024-09-06T14:25:00.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:00.873+0000 7f3e84c18640 1 -- start start 2024-09-06T14:25:00.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:00.874+0000 7f3e84c18640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3e801019c0 0x7f3e8019a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:25:00.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:00.874+0000 7f3e84c18640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3e801073d0 0x7f3e8019ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:25:00.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:00.874+0000 7f3e84c18640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3e80107d50 0x7f3e801a0130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:25:00.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:00.874+0000 7f3e84c18640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f3e801110e0 con 0x7f3e801073d0 2024-09-06T14:25:00.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:00.874+0000 7f3e84c18640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f3e80110f60 con 0x7f3e801019c0 2024-09-06T14:25:00.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:00.874+0000 7f3e84c18640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7f3e80111260 con 0x7f3e80107d50 2024-09-06T14:25:00.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:00.874+0000 7f3e7dd9b640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3e801073d0 0x7f3e8019ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:25:00.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:00.874+0000 7f3e7ed9d640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3e80107d50 0x7f3e801a0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:25:00.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:00.874+0000 7f3e7dd9b640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3e801073d0 0x7f3e8019ad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:55120/0 (socket says 172.21.15.142:55120) 2024-09-06T14:25:00.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:00.874+0000 7f3e7dd9b640 1 -- 172.21.15.142:0/2977727713 learned_addr learned my addr 172.21.15.142:0/2977727713 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:25:00.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:00.874+0000 7f3e7e59c640 1 --2- 172.21.15.142:0/2977727713 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3e801019c0 0x7f3e8019a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:25:00.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:00.874+0000 7f3e7dd9b640 1 -- 172.21.15.142:0/2977727713 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3e80107d50 msgr2=0x7f3e801a0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:25:00.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:00.874+0000 7f3e7dd9b640 1 --2- 172.21.15.142:0/2977727713 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3e80107d50 0x7f3e801a0130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:00.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:00.874+0000 7f3e7dd9b640 1 -- 172.21.15.142:0/2977727713 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3e801019c0 msgr2=0x7f3e8019a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:25:00.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:00.874+0000 7f3e7dd9b640 1 --2- 172.21.15.142:0/2977727713 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3e801019c0 0x7f3e8019a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:00.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:00.874+0000 7f3e7dd9b640 1 -- 172.21.15.142:0/2977727713 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e801a0900 con 0x7f3e801073d0 2024-09-06T14:25:00.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:00.874+0000 7f3e7ed9d640 1 --2- 172.21.15.142:0/2977727713 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3e80107d50 0x7f3e801a0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-06T14:25:00.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:00.874+0000 7f3e7dd9b640 1 --2- 172.21.15.142:0/2977727713 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3e801073d0 0x7f3e8019ad90 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f3e7400d950 tx=0x7f3e7400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:25:00.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:00.874+0000 7f3e6f7fe640 1 -- 172.21.15.142:0/2977727713 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e74017030 con 0x7f3e801073d0 2024-09-06T14:25:00.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:00.874+0000 7f3e84c18640 1 -- 172.21.15.142:0/2977727713 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e801a0bf0 con 0x7f3e801073d0 2024-09-06T14:25:00.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:00.875+0000 7f3e6f7fe640 1 -- 172.21.15.142:0/2977727713 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7f3e74004020 con 0x7f3e801073d0 2024-09-06T14:25:00.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:00.875+0000 7f3e6f7fe640 1 -- 172.21.15.142:0/2977727713 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e740055a0 con 0x7f3e801073d0 2024-09-06T14:25:00.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:00.875+0000 7f3e84c18640 1 -- 172.21.15.142:0/2977727713 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e801a7480 con 0x7f3e801073d0 2024-09-06T14:25:00.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:00.876+0000 7f3e84c18640 1 -- 172.21.15.142:0/2977727713 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e80101da0 con 0x7f3e801073d0 2024-09-06T14:25:00.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:00.876+0000 7f3e6f7fe640 1 -- 172.21.15.142:0/2977727713 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 22) ==== 99844+0+0 (secure 0 0 0) 0x7f3e74009a30 con 0x7f3e801073d0 2024-09-06T14:25:00.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:00.877+0000 7f3e6f7fe640 1 --2- 172.21.15.142:0/2977727713 >> v2:172.21.15.142:6800/2944545612 conn(0x7f3e68077620 0x7f3e68079ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:25:00.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:00.877+0000 7f3e6f7fe640 1 -- 172.21.15.142:0/2977727713 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 6043+0+0 (secure 0 0 0) 0x7f3e74099270 con 0x7f3e801073d0 2024-09-06T14:25:00.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:00.877+0000 7f3e7e59c640 1 --2- 172.21.15.142:0/2977727713 >> v2:172.21.15.142:6800/2944545612 conn(0x7f3e68077620 0x7f3e68079ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:25:00.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:00.877+0000 7f3e7e59c640 1 --2- 172.21.15.142:0/2977727713 >> v2:172.21.15.142:6800/2944545612 conn(0x7f3e68077620 0x7f3e68079ae0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f3e70009530 tx=0x7f3e70009260 comp rx=0 tx=0).ready entity=mgr.14601 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:25:00.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:00.881+0000 7f3e6f7fe640 1 -- 172.21.15.142:0/2977727713 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7f3e74066500 con 0x7f3e801073d0 2024-09-06T14:25:00.958 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:00 smithi142 ceph-mon[29958]: Adjusting osd_memory_target on smithi162 to 3760M 2024-09-06T14:25:00.958 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:00 smithi142 ceph-mon[29958]: pgmap v5: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail 2024-09-06T14:25:00.958 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:00 smithi142 ceph-mon[29958]: mgrmap e22: y(active, since 4s), standbys: x 2024-09-06T14:25:00.959 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:00 smithi142 ceph-mon[37372]: Adjusting osd_memory_target on smithi162 to 3760M 2024-09-06T14:25:00.959 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:00 smithi142 ceph-mon[37372]: pgmap v5: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail 2024-09-06T14:25:00.959 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:00 smithi142 ceph-mon[37372]: mgrmap e22: y(active, since 4s), standbys: x 2024-09-06T14:25:01.005 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:25:01.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:00 smithi162 ceph-mon[33787]: Adjusting osd_memory_target on smithi162 to 3760M 2024-09-06T14:25:01.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:00 smithi162 ceph-mon[33787]: pgmap v5: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail 2024-09-06T14:25:01.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:00 smithi162 ceph-mon[33787]: mgrmap e22: y(active, since 4s), standbys: x 2024-09-06T14:25:01.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:01.061+0000 7f3e84c18640 1 -- 172.21.15.142:0/2977727713 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f3e801a7650 con 0x7f3e801073d0 2024-09-06T14:25:01.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:01.061+0000 7f3e6f7fe640 1 -- 172.21.15.142:0/2977727713 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f3e7406af20 con 0x7f3e801073d0 2024-09-06T14:25:01.058 INFO:teuthology.orchestra.run.smithi142.stdout:107374182431 2024-09-06T14:25:01.060 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:01.063+0000 7f3e84c18640 1 -- 172.21.15.142:0/2977727713 >> v2:172.21.15.142:6800/2944545612 conn(0x7f3e68077620 msgr2=0x7f3e68079ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:25:01.060 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:01.064+0000 7f3e84c18640 1 --2- 172.21.15.142:0/2977727713 >> v2:172.21.15.142:6800/2944545612 conn(0x7f3e68077620 0x7f3e68079ae0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f3e70009530 tx=0x7f3e70009260 comp rx=0 tx=0).stop 2024-09-06T14:25:01.060 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:01.064+0000 7f3e84c18640 1 -- 172.21.15.142:0/2977727713 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3e801073d0 msgr2=0x7f3e8019ad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:25:01.060 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:01.064+0000 7f3e84c18640 1 --2- 172.21.15.142:0/2977727713 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3e801073d0 0x7f3e8019ad90 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f3e7400d950 tx=0x7f3e7400de10 comp rx=0 tx=0).stop 2024-09-06T14:25:01.061 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:01.064+0000 7f3e84c18640 1 -- 172.21.15.142:0/2977727713 shutdown_connections 2024-09-06T14:25:01.061 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:01.064+0000 7f3e84c18640 1 --2- 172.21.15.142:0/2977727713 >> v2:172.21.15.142:6800/2944545612 conn(0x7f3e68077620 0x7f3e68079ae0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:01.061 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:01.064+0000 7f3e84c18640 1 --2- 172.21.15.142:0/2977727713 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3e80107d50 0x7f3e801a0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:01.061 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:01.064+0000 7f3e84c18640 1 --2- 172.21.15.142:0/2977727713 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3e801073d0 0x7f3e8019ad90 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:01.061 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:01.064+0000 7f3e84c18640 1 --2- 172.21.15.142:0/2977727713 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3e801019c0 0x7f3e8019a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:01.061 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:01.064+0000 7f3e84c18640 1 -- 172.21.15.142:0/2977727713 >> 172.21.15.142:0/2977727713 conn(0x7f3e800fd430 msgr2=0x7f3e8006a850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:25:01.061 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:01.064+0000 7f3e84c18640 1 -- 172.21.15.142:0/2977727713 shutdown_connections 2024-09-06T14:25:01.061 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:01.064+0000 7f3e84c18640 1 -- 172.21.15.142:0/2977727713 wait complete. 2024-09-06T14:25:01.425 INFO:teuthology.orchestra.run.smithi142.stdout:55834574888 2024-09-06T14:25:01.426 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-09-06T14:25:02.044 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:01 smithi142 ceph-mon[37372]: from='client.? 172.21.15.142:0/2977727713' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-06T14:25:02.044 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:01 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/2977727713' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-06T14:25:02.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:01 smithi162 ceph-mon[33787]: from='client.? 172.21.15.142:0/2977727713' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-06T14:25:02.072 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182430 got 107374182431 for osd.3 2024-09-06T14:25:02.073 DEBUG:teuthology.parallel:result is None 2024-09-06T14:25:02.073 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:25:02.080 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:25:02.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:02.740+0000 7fea65019640 1 -- 172.21.15.142:0/2231098022 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fea60107d50 msgr2=0x7fea6010e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:25:02.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:02.740+0000 7fea65019640 1 --2- 172.21.15.142:0/2231098022 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fea60107d50 0x7fea6010e550 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fea58009ca0 tx=0x7fea5802a8c0 comp rx=0 tx=0).stop 2024-09-06T14:25:02.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:02.740+0000 7fea65019640 1 -- 172.21.15.142:0/2231098022 shutdown_connections 2024-09-06T14:25:02.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:02.740+0000 7fea65019640 1 --2- 172.21.15.142:0/2231098022 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fea60107d50 0x7fea6010e550 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:02.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:02.740+0000 7fea65019640 1 --2- 172.21.15.142:0/2231098022 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fea601073d0 0x7fea60107810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:02.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:02.740+0000 7fea65019640 1 --2- 172.21.15.142:0/2231098022 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fea601019c0 0x7fea60101da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:02.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:02.740+0000 7fea65019640 1 -- 172.21.15.142:0/2231098022 >> 172.21.15.142:0/2231098022 conn(0x7fea600fd430 msgr2=0x7fea600ff850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:25:02.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:02.740+0000 7fea65019640 1 -- 172.21.15.142:0/2231098022 shutdown_connections 2024-09-06T14:25:02.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:02.740+0000 7fea65019640 1 -- 172.21.15.142:0/2231098022 wait complete. 2024-09-06T14:25:02.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:02.740+0000 7fea65019640 1 Processor -- start 2024-09-06T14:25:02.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:02.741+0000 7fea65019640 1 -- start start 2024-09-06T14:25:02.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:02.741+0000 7fea65019640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fea601019c0 0x7fea601975e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:25:02.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:02.741+0000 7fea65019640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fea601073d0 0x7fea60197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:25:02.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:02.741+0000 7fea65019640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fea60107d50 0x7fea6019cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:25:02.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:02.741+0000 7fea65019640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fea60111120 con 0x7fea60107d50 2024-09-06T14:25:02.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:02.741+0000 7fea65019640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fea60110fa0 con 0x7fea601019c0 2024-09-06T14:25:02.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:02.741+0000 7fea65019640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7fea601112a0 con 0x7fea601073d0 2024-09-06T14:25:02.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:02.741+0000 7fea5f59e640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fea60107d50 0x7fea6019cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:25:02.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:02.741+0000 7fea5e59c640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fea601073d0 0x7fea60197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:25:02.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:02.741+0000 7fea5f59e640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fea60107d50 0x7fea6019cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:55148/0 (socket says 172.21.15.142:55148) 2024-09-06T14:25:02.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:02.741+0000 7fea5f59e640 1 -- 172.21.15.142:0/394142335 learned_addr learned my addr 172.21.15.142:0/394142335 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:25:02.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:02.741+0000 7fea5ed9d640 1 --2- 172.21.15.142:0/394142335 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fea601019c0 0x7fea601975e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:25:02.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:02.741+0000 7fea5e59c640 1 -- 172.21.15.142:0/394142335 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fea601019c0 msgr2=0x7fea601975e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:25:02.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:02.741+0000 7fea5e59c640 1 --2- 172.21.15.142:0/394142335 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fea601019c0 0x7fea601975e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:02.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:02.741+0000 7fea5e59c640 1 -- 172.21.15.142:0/394142335 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fea60107d50 msgr2=0x7fea6019cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:25:02.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:02.741+0000 7fea5e59c640 1 --2- 172.21.15.142:0/394142335 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fea60107d50 0x7fea6019cec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:02.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:02.741+0000 7fea5e59c640 1 -- 172.21.15.142:0/394142335 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea6019d690 con 0x7fea601073d0 2024-09-06T14:25:02.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:02.741+0000 7fea5f59e640 1 --2- 172.21.15.142:0/394142335 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fea60107d50 0x7fea6019cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-06T14:25:02.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:02.741+0000 7fea5e59c640 1 --2- 172.21.15.142:0/394142335 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fea601073d0 0x7fea60197b20 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fea5400eb00 tx=0x7fea5400ef00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:25:02.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:02.742+0000 7fea47fff640 1 -- 172.21.15.142:0/394142335 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea5400bb40 con 0x7fea601073d0 2024-09-06T14:25:02.747 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:02.742+0000 7fea47fff640 1 -- 172.21.15.142:0/394142335 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7fea5400bce0 con 0x7fea601073d0 2024-09-06T14:25:02.747 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:02.742+0000 7fea65019640 1 -- 172.21.15.142:0/394142335 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea6019d980 con 0x7fea601073d0 2024-09-06T14:25:02.747 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:02.742+0000 7fea47fff640 1 -- 172.21.15.142:0/394142335 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea54003da0 con 0x7fea601073d0 2024-09-06T14:25:02.747 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:02.742+0000 7fea65019640 1 -- 172.21.15.142:0/394142335 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fea6006a860 con 0x7fea601073d0 2024-09-06T14:25:02.747 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:02.743+0000 7fea65019640 1 -- 172.21.15.142:0/394142335 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea60101da0 con 0x7fea601073d0 2024-09-06T14:25:02.747 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:02.743+0000 7fea47fff640 1 -- 172.21.15.142:0/394142335 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 22) ==== 99844+0+0 (secure 0 0 0) 0x7fea54017410 con 0x7fea601073d0 2024-09-06T14:25:02.747 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:02.744+0000 7fea47fff640 1 --2- 172.21.15.142:0/394142335 >> v2:172.21.15.142:6800/2944545612 conn(0x7fea48077620 0x7fea48079ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:25:02.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:02.744+0000 7fea47fff640 1 -- 172.21.15.142:0/394142335 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(62..62 src has 1..62) ==== 6043+0+0 (secure 0 0 0) 0x7fea5409ae90 con 0x7fea601073d0 2024-09-06T14:25:02.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:02.744+0000 7fea5ed9d640 1 --2- 172.21.15.142:0/394142335 >> v2:172.21.15.142:6800/2944545612 conn(0x7fea48077620 0x7fea48079ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:25:02.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:02.744+0000 7fea5ed9d640 1 --2- 172.21.15.142:0/394142335 >> v2:172.21.15.142:6800/2944545612 conn(0x7fea48077620 0x7fea48079ae0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fea50009990 tx=0x7fea50009110 comp rx=0 tx=0).ready entity=mgr.14601 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:25:02.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:02.748+0000 7fea47fff640 1 -- 172.21.15.142:0/394142335 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7fea5406cb40 con 0x7fea601073d0 2024-09-06T14:25:02.906 INFO:teuthology.orchestra.run.smithi142.stdout:128849018906 2024-09-06T14:25:02.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:02.908+0000 7fea65019640 1 -- 172.21.15.142:0/394142335 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7fea6006b280 con 0x7fea601073d0 2024-09-06T14:25:02.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:02.909+0000 7fea47fff640 1 -- 172.21.15.142:0/394142335 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fea5405f070 con 0x7fea601073d0 2024-09-06T14:25:02.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:02.911+0000 7fea65019640 1 -- 172.21.15.142:0/394142335 >> v2:172.21.15.142:6800/2944545612 conn(0x7fea48077620 msgr2=0x7fea48079ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:25:02.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:02.911+0000 7fea65019640 1 --2- 172.21.15.142:0/394142335 >> v2:172.21.15.142:6800/2944545612 conn(0x7fea48077620 0x7fea48079ae0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fea50009990 tx=0x7fea50009110 comp rx=0 tx=0).stop 2024-09-06T14:25:02.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:02.911+0000 7fea65019640 1 -- 172.21.15.142:0/394142335 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fea601073d0 msgr2=0x7fea60197b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:25:02.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:02.911+0000 7fea65019640 1 --2- 172.21.15.142:0/394142335 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fea601073d0 0x7fea60197b20 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fea5400eb00 tx=0x7fea5400ef00 comp rx=0 tx=0).stop 2024-09-06T14:25:02.908 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:02 smithi142 ceph-mon[29958]: pgmap v6: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 15 op/s 2024-09-06T14:25:02.908 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:02 smithi142 ceph-mon[37372]: pgmap v6: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 15 op/s 2024-09-06T14:25:02.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:02.911+0000 7fea65019640 1 -- 172.21.15.142:0/394142335 shutdown_connections 2024-09-06T14:25:02.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:02.911+0000 7fea65019640 1 --2- 172.21.15.142:0/394142335 >> v2:172.21.15.142:6800/2944545612 conn(0x7fea48077620 0x7fea48079ae0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:02.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:02.911+0000 7fea65019640 1 --2- 172.21.15.142:0/394142335 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fea60107d50 0x7fea6019cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:02.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:02.911+0000 7fea65019640 1 --2- 172.21.15.142:0/394142335 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fea601073d0 0x7fea60197b20 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:02.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:02.911+0000 7fea65019640 1 --2- 172.21.15.142:0/394142335 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fea601019c0 0x7fea601975e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:02.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:02.911+0000 7fea65019640 1 -- 172.21.15.142:0/394142335 >> 172.21.15.142:0/394142335 conn(0x7fea600fd430 msgr2=0x7fea600ff820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:25:02.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:02.911+0000 7fea65019640 1 -- 172.21.15.142:0/394142335 shutdown_connections 2024-09-06T14:25:02.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:02.911+0000 7fea65019640 1 -- 172.21.15.142:0/394142335 wait complete. 2024-09-06T14:25:02.941 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:02.937+0000 7f11fad81640 1 -- 172.21.15.142:0/1575780103 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f11f4107e40 msgr2=0x7f11f410e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:25:02.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:02.937+0000 7f11fad81640 1 --2- 172.21.15.142:0/1575780103 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f11f4107e40 0x7f11f410e570 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f11ec009ca0 tx=0x7f11ec02a8c0 comp rx=0 tx=0).stop 2024-09-06T14:25:02.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:02.937+0000 7f11fad81640 1 -- 172.21.15.142:0/1575780103 shutdown_connections 2024-09-06T14:25:02.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:02.937+0000 7f11fad81640 1 --2- 172.21.15.142:0/1575780103 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f11f4107e40 0x7f11f410e570 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:02.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:02.937+0000 7f11fad81640 1 --2- 172.21.15.142:0/1575780103 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f11f41074c0 0x7f11f4107900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:02.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:02.937+0000 7f11fad81640 1 --2- 172.21.15.142:0/1575780103 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f11f41026d0 0x7f11f4102ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:02.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:02.937+0000 7f11fad81640 1 -- 172.21.15.142:0/1575780103 >> 172.21.15.142:0/1575780103 conn(0x7f11f4076c00 msgr2=0x7f11f4077010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:25:02.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:02.937+0000 7f11fad81640 1 -- 172.21.15.142:0/1575780103 shutdown_connections 2024-09-06T14:25:02.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:02.937+0000 7f11fad81640 1 -- 172.21.15.142:0/1575780103 wait complete. 2024-09-06T14:25:02.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:02.938+0000 7f11fad81640 1 Processor -- start 2024-09-06T14:25:02.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:02.938+0000 7f11fad81640 1 -- start start 2024-09-06T14:25:02.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:02.938+0000 7f11fad81640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f11f41026d0 0x7f11f4074860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:25:02.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:02.938+0000 7f11fad81640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f11f41074c0 0x7f11f406f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:25:02.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:02.938+0000 7f11fad81640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f11f4107e40 0x7f11f406fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:25:02.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:02.939+0000 7f11f8b1d640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f11f41026d0 0x7f11f4074860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:25:02.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:02.939+0000 7f11fad81640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f11f4111110 con 0x7f11f4107e40 2024-09-06T14:25:02.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:02.939+0000 7f11fad81640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f11f4110f90 con 0x7f11f41074c0 2024-09-06T14:25:02.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:02.939+0000 7f11fad81640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7f11f4111290 con 0x7f11f41026d0 2024-09-06T14:25:02.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:02.939+0000 7f11f8b1d640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f11f41026d0 0x7f11f4074860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:45032/0 (socket says 172.21.15.142:45032) 2024-09-06T14:25:02.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:02.939+0000 7f11f8b1d640 1 -- 172.21.15.142:0/2462305601 learned_addr learned my addr 172.21.15.142:0/2462305601 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:25:02.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:02.939+0000 7f11f931e640 1 --2- 172.21.15.142:0/2462305601 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f11f4107e40 0x7f11f406fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:25:02.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:02.939+0000 7f11ebfff640 1 --2- 172.21.15.142:0/2462305601 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f11f41074c0 0x7f11f406f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:25:02.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:02.939+0000 7f11f8b1d640 1 -- 172.21.15.142:0/2462305601 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f11f41074c0 msgr2=0x7f11f406f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:25:02.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:02.939+0000 7f11f8b1d640 1 --2- 172.21.15.142:0/2462305601 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f11f41074c0 0x7f11f406f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:02.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:02.939+0000 7f11f8b1d640 1 -- 172.21.15.142:0/2462305601 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f11f4107e40 msgr2=0x7f11f406fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:25:02.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:02.939+0000 7f11f8b1d640 1 --2- 172.21.15.142:0/2462305601 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f11f4107e40 0x7f11f406fd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:02.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:02.939+0000 7f11f8b1d640 1 -- 172.21.15.142:0/2462305601 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11f4070310 con 0x7f11f41026d0 2024-09-06T14:25:02.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:02.939+0000 7f11f8b1d640 1 --2- 172.21.15.142:0/2462305601 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f11f41026d0 0x7f11f4074860 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f11f000ea60 tx=0x7f11f000ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:25:02.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:02.939+0000 7f11e9ffb640 1 -- 172.21.15.142:0/2462305601 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11f000cde0 con 0x7f11f41026d0 2024-09-06T14:25:02.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:02.939+0000 7f11e9ffb640 1 -- 172.21.15.142:0/2462305601 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7f11f00035f0 con 0x7f11f41026d0 2024-09-06T14:25:02.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:02.939+0000 7f11fad81640 1 -- 172.21.15.142:0/2462305601 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11f4070600 con 0x7f11f41026d0 2024-09-06T14:25:02.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:02.939+0000 7f11e9ffb640 1 -- 172.21.15.142:0/2462305601 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11f00044a0 con 0x7f11f41026d0 2024-09-06T14:25:02.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:02.940+0000 7f11fad81640 1 -- 172.21.15.142:0/2462305601 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f11f406d920 con 0x7f11f41026d0 2024-09-06T14:25:02.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:02.941+0000 7f11fad81640 1 -- 172.21.15.142:0/2462305601 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11f4102ab0 con 0x7f11f41026d0 2024-09-06T14:25:02.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:02.942+0000 7f11e9ffb640 1 -- 172.21.15.142:0/2462305601 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 22) ==== 99844+0+0 (secure 0 0 0) 0x7f11f0004000 con 0x7f11f41026d0 2024-09-06T14:25:02.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:02.942+0000 7f11e9ffb640 1 --2- 172.21.15.142:0/2462305601 >> v2:172.21.15.142:6800/2944545612 conn(0x7f11dc077620 0x7f11dc079ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:25:02.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:02.942+0000 7f11ebfff640 1 --2- 172.21.15.142:0/2462305601 >> v2:172.21.15.142:6800/2944545612 conn(0x7f11dc077620 0x7f11dc079ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:25:02.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:02.942+0000 7f11e9ffb640 1 -- 172.21.15.142:0/2462305601 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(62..62 src has 1..62) ==== 6043+0+0 (secure 0 0 0) 0x7f11f0099dd0 con 0x7f11f41026d0 2024-09-06T14:25:02.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:02.943+0000 7f11ebfff640 1 --2- 172.21.15.142:0/2462305601 >> v2:172.21.15.142:6800/2944545612 conn(0x7f11dc077620 0x7f11dc079ae0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f11e4005fd0 tx=0x7f11e4005850 comp rx=0 tx=0).ready entity=mgr.14601 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:25:02.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:02.947+0000 7f11e9ffb640 1 -- 172.21.15.142:0/2462305601 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7f11f0062e50 con 0x7f11f41026d0 2024-09-06T14:25:03.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:02 smithi162 ceph-mon[33787]: pgmap v6: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 15 op/s 2024-09-06T14:25:03.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:03.105+0000 7f11fad81640 1 -- 172.21.15.142:0/2462305601 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f11f4109b10 con 0x7f11f41026d0 2024-09-06T14:25:03.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:03.105+0000 7f11e9ffb640 1 -- 172.21.15.142:0/2462305601 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f11f0066fc0 con 0x7f11f41026d0 2024-09-06T14:25:03.102 INFO:teuthology.orchestra.run.smithi142.stdout:77309411364 2024-09-06T14:25:03.104 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:03.107+0000 7f11fad81640 1 -- 172.21.15.142:0/2462305601 >> v2:172.21.15.142:6800/2944545612 conn(0x7f11dc077620 msgr2=0x7f11dc079ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:25:03.104 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:03.107+0000 7f11fad81640 1 --2- 172.21.15.142:0/2462305601 >> v2:172.21.15.142:6800/2944545612 conn(0x7f11dc077620 0x7f11dc079ae0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f11e4005fd0 tx=0x7f11e4005850 comp rx=0 tx=0).stop 2024-09-06T14:25:03.104 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:03.107+0000 7f11fad81640 1 -- 172.21.15.142:0/2462305601 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f11f41026d0 msgr2=0x7f11f4074860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:25:03.104 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:03.107+0000 7f11fad81640 1 --2- 172.21.15.142:0/2462305601 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f11f41026d0 0x7f11f4074860 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f11f000ea60 tx=0x7f11f000ef20 comp rx=0 tx=0).stop 2024-09-06T14:25:03.105 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:03.107+0000 7f11fad81640 1 -- 172.21.15.142:0/2462305601 shutdown_connections 2024-09-06T14:25:03.105 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:03.107+0000 7f11fad81640 1 --2- 172.21.15.142:0/2462305601 >> v2:172.21.15.142:6800/2944545612 conn(0x7f11dc077620 0x7f11dc079ae0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:03.105 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:03.107+0000 7f11fad81640 1 --2- 172.21.15.142:0/2462305601 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f11f4107e40 0x7f11f406fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:03.105 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:03.107+0000 7f11fad81640 1 --2- 172.21.15.142:0/2462305601 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f11f41074c0 0x7f11f406f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:03.105 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:03.107+0000 7f11fad81640 1 --2- 172.21.15.142:0/2462305601 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f11f41026d0 0x7f11f4074860 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:03.105 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:03.107+0000 7f11fad81640 1 -- 172.21.15.142:0/2462305601 >> 172.21.15.142:0/2462305601 conn(0x7f11f4076c00 msgr2=0x7f11f4100150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:25:03.105 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:03.107+0000 7f11fad81640 1 -- 172.21.15.142:0/2462305601 shutdown_connections 2024-09-06T14:25:03.105 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:03.107+0000 7f11fad81640 1 -- 172.21.15.142:0/2462305601 wait complete. 2024-09-06T14:25:03.878 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411362 got 77309411364 for osd.2 2024-09-06T14:25:03.878 DEBUG:teuthology.parallel:result is None 2024-09-06T14:25:03.958 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:03 smithi142 ceph-mon[37372]: from='client.? 172.21.15.142:0/394142335' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-06T14:25:03.958 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:03 smithi142 ceph-mon[37372]: from='client.? 172.21.15.142:0/2462305601' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-06T14:25:03.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:03 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/394142335' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-06T14:25:03.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:03 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/2462305601' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-06T14:25:04.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:03 smithi162 ceph-mon[33787]: from='client.? 172.21.15.142:0/394142335' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-06T14:25:04.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:03 smithi162 ceph-mon[33787]: from='client.? 172.21.15.142:0/2462305601' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-06T14:25:04.391 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:25:04.392 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:25:04.431 INFO:tasks.cephadm.ceph_manager.ceph:need seq 128849018904 got 128849018906 for osd.4 2024-09-06T14:25:04.432 DEBUG:teuthology.parallel:result is None 2024-09-06T14:25:04.439 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:25:04.706 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:25:04 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: ::ffff:172.21.15.162 - - [06/Sep/2024:14:25:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-06T14:25:04.958 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:04 smithi142 ceph-mon[29958]: pgmap v7: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-06T14:25:04.959 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:04 smithi142 ceph-mon[37372]: pgmap v7: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-06T14:25:05.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:04 smithi162 ceph-mon[33787]: pgmap v7: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-06T14:25:05.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.157+0000 7fd35e535640 1 -- 172.21.15.142:0/2731023250 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd358108e70 msgr2=0x7fd35810f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:25:05.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.157+0000 7fd35e535640 1 --2- 172.21.15.142:0/2731023250 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd358108e70 0x7fd35810f670 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fd350009ca0 tx=0x7fd35002a8c0 comp rx=0 tx=0).stop 2024-09-06T14:25:05.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.157+0000 7fd35e535640 1 -- 172.21.15.142:0/2731023250 shutdown_connections 2024-09-06T14:25:05.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.157+0000 7fd35e535640 1 --2- 172.21.15.142:0/2731023250 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd358108e70 0x7fd35810f670 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:05.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.157+0000 7fd35e535640 1 --2- 172.21.15.142:0/2731023250 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd3581084f0 0x7fd358108930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:05.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.157+0000 7fd35e535640 1 --2- 172.21.15.142:0/2731023250 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd358102ae0 0x7fd358102ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:05.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.157+0000 7fd35e535640 1 -- 172.21.15.142:0/2731023250 >> 172.21.15.142:0/2731023250 conn(0x7fd3580fe530 msgr2=0x7fd358100950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:25:05.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.157+0000 7fd35e535640 1 -- 172.21.15.142:0/2731023250 shutdown_connections 2024-09-06T14:25:05.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.157+0000 7fd35e535640 1 -- 172.21.15.142:0/2731023250 wait complete. 2024-09-06T14:25:05.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.158+0000 7fd35e535640 1 Processor -- start 2024-09-06T14:25:05.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.158+0000 7fd35e535640 1 -- start start 2024-09-06T14:25:05.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.158+0000 7fd35e535640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd358102ae0 0x7fd3581986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:25:05.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.158+0000 7fd35e535640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd3581084f0 0x7fd358198bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:25:05.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.158+0000 7fd35e535640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd358108e70 0x7fd35819df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:25:05.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.158+0000 7fd35e535640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fd358112200 con 0x7fd358108e70 2024-09-06T14:25:05.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.158+0000 7fd35e535640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd358112080 con 0x7fd3581084f0 2024-09-06T14:25:05.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.158+0000 7fd35e535640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7fd358112380 con 0x7fd358102ae0 2024-09-06T14:25:05.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.158+0000 7fd357fff640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd358102ae0 0x7fd3581986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:25:05.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.158+0000 7fd35cad2640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd358108e70 0x7fd35819df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:25:05.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.158+0000 7fd357fff640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd358102ae0 0x7fd3581986b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:45060/0 (socket says 172.21.15.142:45060) 2024-09-06T14:25:05.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.158+0000 7fd357fff640 1 -- 172.21.15.142:0/487835571 learned_addr learned my addr 172.21.15.142:0/487835571 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:25:05.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.158+0000 7fd3577fe640 1 --2- 172.21.15.142:0/487835571 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd3581084f0 0x7fd358198bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:25:05.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.158+0000 7fd357fff640 1 -- 172.21.15.142:0/487835571 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd3581084f0 msgr2=0x7fd358198bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:25:05.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.158+0000 7fd357fff640 1 --2- 172.21.15.142:0/487835571 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd3581084f0 0x7fd358198bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:05.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.158+0000 7fd357fff640 1 -- 172.21.15.142:0/487835571 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd358108e70 msgr2=0x7fd35819df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:25:05.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.158+0000 7fd357fff640 1 --2- 172.21.15.142:0/487835571 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd358108e70 0x7fd35819df90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:05.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.159+0000 7fd357fff640 1 -- 172.21.15.142:0/487835571 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd35819e760 con 0x7fd358102ae0 2024-09-06T14:25:05.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.159+0000 7fd35cad2640 1 --2- 172.21.15.142:0/487835571 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd358108e70 0x7fd35819df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-06T14:25:05.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.159+0000 7fd357fff640 1 --2- 172.21.15.142:0/487835571 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd358102ae0 0x7fd3581986b0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd34800ea60 tx=0x7fd34800ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:25:05.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.159+0000 7fd3557fa640 1 -- 172.21.15.142:0/487835571 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd34800cde0 con 0x7fd358102ae0 2024-09-06T14:25:05.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.159+0000 7fd35e535640 1 -- 172.21.15.142:0/487835571 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd35819ea50 con 0x7fd358102ae0 2024-09-06T14:25:05.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.159+0000 7fd3557fa640 1 -- 172.21.15.142:0/487835571 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7fd3480035f0 con 0x7fd358102ae0 2024-09-06T14:25:05.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.159+0000 7fd3557fa640 1 -- 172.21.15.142:0/487835571 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3480044a0 con 0x7fd358102ae0 2024-09-06T14:25:05.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.159+0000 7fd35e535640 1 -- 172.21.15.142:0/487835571 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3581a52e0 con 0x7fd358102ae0 2024-09-06T14:25:05.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.160+0000 7fd35e535640 1 -- 172.21.15.142:0/487835571 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd358102ec0 con 0x7fd358102ae0 2024-09-06T14:25:05.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.161+0000 7fd3557fa640 1 -- 172.21.15.142:0/487835571 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 22) ==== 99844+0+0 (secure 0 0 0) 0x7fd348004000 con 0x7fd358102ae0 2024-09-06T14:25:05.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.161+0000 7fd3557fa640 1 --2- 172.21.15.142:0/487835571 >> v2:172.21.15.142:6800/2944545612 conn(0x7fd340077620 0x7fd340079ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:25:05.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.161+0000 7fd3557fa640 1 -- 172.21.15.142:0/487835571 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(62..62 src has 1..62) ==== 6043+0+0 (secure 0 0 0) 0x7fd348099c70 con 0x7fd358102ae0 2024-09-06T14:25:05.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.161+0000 7fd3577fe640 1 --2- 172.21.15.142:0/487835571 >> v2:172.21.15.142:6800/2944545612 conn(0x7fd340077620 0x7fd340079ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:25:05.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.162+0000 7fd3577fe640 1 --2- 172.21.15.142:0/487835571 >> v2:172.21.15.142:6800/2944545612 conn(0x7fd340077620 0x7fd340079ae0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fd358108080 tx=0x7fd34c00b390 comp rx=0 tx=0).ready entity=mgr.14601 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:25:05.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.165+0000 7fd3557fa640 1 -- 172.21.15.142:0/487835571 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7fd348066f00 con 0x7fd358102ae0 2024-09-06T14:25:05.339 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.342+0000 7fd35e535640 1 -- 172.21.15.142:0/487835571 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7fd3581a54b0 con 0x7fd358102ae0 2024-09-06T14:25:05.339 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.342+0000 7fd3557fa640 1 -- 172.21.15.142:0/487835571 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fd34806b920 con 0x7fd358102ae0 2024-09-06T14:25:05.339 INFO:teuthology.orchestra.run.smithi142.stdout:180388626448 2024-09-06T14:25:05.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.344+0000 7fd35e535640 1 -- 172.21.15.142:0/487835571 >> v2:172.21.15.142:6800/2944545612 conn(0x7fd340077620 msgr2=0x7fd340079ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:25:05.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.344+0000 7fd35e535640 1 --2- 172.21.15.142:0/487835571 >> v2:172.21.15.142:6800/2944545612 conn(0x7fd340077620 0x7fd340079ae0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fd358108080 tx=0x7fd34c00b390 comp rx=0 tx=0).stop 2024-09-06T14:25:05.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.345+0000 7fd35e535640 1 -- 172.21.15.142:0/487835571 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd358102ae0 msgr2=0x7fd3581986b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:25:05.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.345+0000 7fd35e535640 1 --2- 172.21.15.142:0/487835571 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd358102ae0 0x7fd3581986b0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd34800ea60 tx=0x7fd34800ef20 comp rx=0 tx=0).stop 2024-09-06T14:25:05.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.345+0000 7fd35e535640 1 -- 172.21.15.142:0/487835571 shutdown_connections 2024-09-06T14:25:05.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.345+0000 7fd35e535640 1 --2- 172.21.15.142:0/487835571 >> v2:172.21.15.142:6800/2944545612 conn(0x7fd340077620 0x7fd340079ae0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:05.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.345+0000 7fd35e535640 1 --2- 172.21.15.142:0/487835571 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd358108e70 0x7fd35819df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:05.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.345+0000 7fd35e535640 1 --2- 172.21.15.142:0/487835571 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd3581084f0 0x7fd358198bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:05.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.345+0000 7fd35e535640 1 --2- 172.21.15.142:0/487835571 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd358102ae0 0x7fd3581986b0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:05.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.345+0000 7fd35e535640 1 -- 172.21.15.142:0/487835571 >> 172.21.15.142:0/487835571 conn(0x7fd3580fe530 msgr2=0x7fd3580ffd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:25:05.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.345+0000 7fd35e535640 1 -- 172.21.15.142:0/487835571 shutdown_connections 2024-09-06T14:25:05.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.345+0000 7fd35e535640 1 -- 172.21.15.142:0/487835571 wait complete. 2024-09-06T14:25:05.441 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.426+0000 7f7d5d989640 1 -- 172.21.15.142:0/1214927238 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d58103860 msgr2=0x7f7d58103c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:25:05.441 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.426+0000 7f7d5d989640 1 --2- 172.21.15.142:0/1214927238 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d58103860 0x7f7d58103c40 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f7d48009900 tx=0x7f7d4802a410 comp rx=0 tx=0).stop 2024-09-06T14:25:05.441 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.426+0000 7f7d5d989640 1 -- 172.21.15.142:0/1214927238 shutdown_connections 2024-09-06T14:25:05.441 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.426+0000 7f7d5d989640 1 --2- 172.21.15.142:0/1214927238 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7d5810ce10 0x7f7d5810f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:05.441 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.426+0000 7f7d5d989640 1 --2- 172.21.15.142:0/1214927238 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7d58104260 0x7f7d5810c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:05.441 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.426+0000 7f7d5d989640 1 --2- 172.21.15.142:0/1214927238 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d58103860 0x7f7d58103c40 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:05.441 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.426+0000 7f7d5d989640 1 -- 172.21.15.142:0/1214927238 >> 172.21.15.142:0/1214927238 conn(0x7f7d580fd430 msgr2=0x7f7d580ff850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:25:05.441 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.427+0000 7f7d5d989640 1 -- 172.21.15.142:0/1214927238 shutdown_connections 2024-09-06T14:25:05.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.427+0000 7f7d5d989640 1 -- 172.21.15.142:0/1214927238 wait complete. 2024-09-06T14:25:05.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.427+0000 7f7d5d989640 1 Processor -- start 2024-09-06T14:25:05.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.427+0000 7f7d5d989640 1 -- start start 2024-09-06T14:25:05.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.428+0000 7f7d5d989640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7d58103860 0x7f7d581986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:25:05.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.428+0000 7f7d5d989640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d58104260 0x7f7d58198c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:25:05.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.428+0000 7f7d5d989640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7d5810ce10 0x7f7d5819dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:25:05.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.428+0000 7f7d5d989640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f7d581121a0 con 0x7f7d5810ce10 2024-09-06T14:25:05.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.428+0000 7f7d5d989640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7d58112020 con 0x7f7d58103860 2024-09-06T14:25:05.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.428+0000 7f7d5d989640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7f7d58112320 con 0x7f7d58104260 2024-09-06T14:25:05.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.428+0000 7f7d4e7fc640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d58104260 0x7f7d58198c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:25:05.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.428+0000 7f7d577fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7d5810ce10 0x7f7d5819dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:25:05.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.428+0000 7f7d4e7fc640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d58104260 0x7f7d58198c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:45086/0 (socket says 172.21.15.142:45086) 2024-09-06T14:25:05.443 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.428+0000 7f7d4e7fc640 1 -- 172.21.15.142:0/768730315 learned_addr learned my addr 172.21.15.142:0/768730315 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:25:05.443 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.428+0000 7f7d577fe640 1 -- 172.21.15.142:0/768730315 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d58104260 msgr2=0x7f7d58198c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:25:05.443 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.428+0000 7f7d56ffd640 1 --2- 172.21.15.142:0/768730315 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7d58103860 0x7f7d581986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:25:05.443 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.428+0000 7f7d577fe640 1 --2- 172.21.15.142:0/768730315 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d58104260 0x7f7d58198c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:05.443 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.428+0000 7f7d577fe640 1 -- 172.21.15.142:0/768730315 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7d58103860 msgr2=0x7f7d581986e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:25:05.443 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.428+0000 7f7d577fe640 1 --2- 172.21.15.142:0/768730315 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7d58103860 0x7f7d581986e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:05.443 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.428+0000 7f7d577fe640 1 -- 172.21.15.142:0/768730315 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d5819e790 con 0x7f7d5810ce10 2024-09-06T14:25:05.443 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.428+0000 7f7d4e7fc640 1 --2- 172.21.15.142:0/768730315 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d58104260 0x7f7d58198c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-06T14:25:05.443 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.428+0000 7f7d56ffd640 1 --2- 172.21.15.142:0/768730315 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7d58103860 0x7f7d581986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-06T14:25:05.443 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.428+0000 7f7d577fe640 1 --2- 172.21.15.142:0/768730315 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7d5810ce10 0x7f7d5819dfc0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f7d5000ec50 tx=0x7f7d5000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:25:05.443 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.428+0000 7f7d5c987640 1 -- 172.21.15.142:0/768730315 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d50003b70 con 0x7f7d5810ce10 2024-09-06T14:25:05.443 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.428+0000 7f7d5d989640 1 -- 172.21.15.142:0/768730315 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d5819ea80 con 0x7f7d5810ce10 2024-09-06T14:25:05.443 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.428+0000 7f7d5c987640 1 -- 172.21.15.142:0/768730315 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7f7d50003d10 con 0x7f7d5810ce10 2024-09-06T14:25:05.444 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.429+0000 7f7d5c987640 1 -- 172.21.15.142:0/768730315 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d50010650 con 0x7f7d5810ce10 2024-09-06T14:25:05.444 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.429+0000 7f7d5d989640 1 -- 172.21.15.142:0/768730315 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d581a5140 con 0x7f7d5810ce10 2024-09-06T14:25:05.444 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.429+0000 7f7d5d989640 1 -- 172.21.15.142:0/768730315 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d58103c40 con 0x7f7d5810ce10 2024-09-06T14:25:05.444 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.430+0000 7f7d5c987640 1 -- 172.21.15.142:0/768730315 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 22) ==== 99844+0+0 (secure 0 0 0) 0x7f7d50004020 con 0x7f7d5810ce10 2024-09-06T14:25:05.444 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.431+0000 7f7d5c987640 1 --2- 172.21.15.142:0/768730315 >> v2:172.21.15.142:6800/2944545612 conn(0x7f7d44077620 0x7f7d44079ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:25:05.444 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.431+0000 7f7d56ffd640 1 --2- 172.21.15.142:0/768730315 >> v2:172.21.15.142:6800/2944545612 conn(0x7f7d44077620 0x7f7d44079ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:25:05.444 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.431+0000 7f7d5c987640 1 -- 172.21.15.142:0/768730315 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 6043+0+0 (secure 0 0 0) 0x7f7d50099e90 con 0x7f7d5810ce10 2024-09-06T14:25:05.444 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.431+0000 7f7d56ffd640 1 --2- 172.21.15.142:0/768730315 >> v2:172.21.15.142:6800/2944545612 conn(0x7f7d44077620 0x7f7d44079ae0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f7d48000c00 tx=0x7f7d48031400 comp rx=0 tx=0).ready entity=mgr.14601 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:25:05.444 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.434+0000 7f7d5c987640 1 -- 172.21.15.142:0/768730315 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7f7d50062fb0 con 0x7f7d5810ce10 2024-09-06T14:25:05.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.588+0000 7f7d5d989640 1 -- 172.21.15.142:0/768730315 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f7d58107570 con 0x7f7d5810ce10 2024-09-06T14:25:05.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.589+0000 7f7d5c987640 1 -- 172.21.15.142:0/768730315 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f7d50067120 con 0x7f7d5810ce10 2024-09-06T14:25:05.586 INFO:teuthology.orchestra.run.smithi142.stdout:206158430219 2024-09-06T14:25:05.588 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.591+0000 7f7d5d989640 1 -- 172.21.15.142:0/768730315 >> v2:172.21.15.142:6800/2944545612 conn(0x7f7d44077620 msgr2=0x7f7d44079ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:25:05.588 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.591+0000 7f7d5d989640 1 --2- 172.21.15.142:0/768730315 >> v2:172.21.15.142:6800/2944545612 conn(0x7f7d44077620 0x7f7d44079ae0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f7d48000c00 tx=0x7f7d48031400 comp rx=0 tx=0).stop 2024-09-06T14:25:05.588 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.591+0000 7f7d5d989640 1 -- 172.21.15.142:0/768730315 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7d5810ce10 msgr2=0x7f7d5819dfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:25:05.588 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.591+0000 7f7d5d989640 1 --2- 172.21.15.142:0/768730315 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7d5810ce10 0x7f7d5819dfc0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f7d5000ec50 tx=0x7f7d5000c6a0 comp rx=0 tx=0).stop 2024-09-06T14:25:05.588 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.591+0000 7f7d5d989640 1 -- 172.21.15.142:0/768730315 shutdown_connections 2024-09-06T14:25:05.588 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.591+0000 7f7d5d989640 1 --2- 172.21.15.142:0/768730315 >> v2:172.21.15.142:6800/2944545612 conn(0x7f7d44077620 0x7f7d44079ae0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:05.588 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.591+0000 7f7d5d989640 1 --2- 172.21.15.142:0/768730315 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7d5810ce10 0x7f7d5819dfc0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:05.588 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.591+0000 7f7d5d989640 1 --2- 172.21.15.142:0/768730315 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d58104260 0x7f7d58198c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:05.588 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.591+0000 7f7d5d989640 1 --2- 172.21.15.142:0/768730315 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7d58103860 0x7f7d581986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:05.588 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.591+0000 7f7d5d989640 1 -- 172.21.15.142:0/768730315 >> 172.21.15.142:0/768730315 conn(0x7f7d580fd430 msgr2=0x7f7d58104740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:25:05.588 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.591+0000 7f7d5d989640 1 -- 172.21.15.142:0/768730315 shutdown_connections 2024-09-06T14:25:05.589 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:05.591+0000 7f7d5d989640 1 -- 172.21.15.142:0/768730315 wait complete. 2024-09-06T14:25:05.928 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:05 smithi142 ceph-mon[37372]: from='client.? 172.21.15.142:0/487835571' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-06T14:25:05.929 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:05 smithi142 ceph-mon[37372]: from='client.? 172.21.15.142:0/768730315' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-06T14:25:05.929 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:05 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/487835571' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-06T14:25:05.929 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:05 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/768730315' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-06T14:25:06.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:05 smithi162 ceph-mon[33787]: from='client.? 172.21.15.142:0/487835571' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-06T14:25:06.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:05 smithi162 ceph-mon[33787]: from='client.? 172.21.15.142:0/768730315' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-06T14:25:06.633 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.609+0000 7f274e5fb640 1 -- 172.21.15.142:0/3395782755 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f274810ce70 msgr2=0x7f274810f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:25:06.633 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.609+0000 7f274e5fb640 1 --2- 172.21.15.142:0/3395782755 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f274810ce70 0x7f274810f280 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f2740009ca0 tx=0x7f274002a8c0 comp rx=0 tx=0).stop 2024-09-06T14:25:06.633 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.609+0000 7f274e5fb640 1 -- 172.21.15.142:0/3395782755 shutdown_connections 2024-09-06T14:25:06.633 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.609+0000 7f274e5fb640 1 --2- 172.21.15.142:0/3395782755 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f274810ce70 0x7f274810f280 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:06.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.609+0000 7f274e5fb640 1 --2- 172.21.15.142:0/3395782755 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f27481042c0 0x7f274810c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:06.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.609+0000 7f274e5fb640 1 --2- 172.21.15.142:0/3395782755 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f27481038c0 0x7f2748103ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:06.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.609+0000 7f274e5fb640 1 -- 172.21.15.142:0/3395782755 >> 172.21.15.142:0/3395782755 conn(0x7f27480fd430 msgr2=0x7f27480ff850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:25:06.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.609+0000 7f274e5fb640 1 -- 172.21.15.142:0/3395782755 shutdown_connections 2024-09-06T14:25:06.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.609+0000 7f274e5fb640 1 -- 172.21.15.142:0/3395782755 wait complete. 2024-09-06T14:25:06.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.610+0000 7f274e5fb640 1 Processor -- start 2024-09-06T14:25:06.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.610+0000 7f274e5fb640 1 -- start start 2024-09-06T14:25:06.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.610+0000 7f274e5fb640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f27481038c0 0x7f27481985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:25:06.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.610+0000 7f274e5fb640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f27481042c0 0x7f2748198b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:25:06.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.610+0000 7f274e5fb640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f274810ce70 0x7f274819ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:25:06.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.611+0000 7f2747fff640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f27481038c0 0x7f27481985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:25:06.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.611+0000 7f274e5fb640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f27481121a0 con 0x7f274810ce70 2024-09-06T14:25:06.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.611+0000 7f274e5fb640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f2748112020 con 0x7f27481042c0 2024-09-06T14:25:06.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.611+0000 7f274e5fb640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7f2748112320 con 0x7f27481038c0 2024-09-06T14:25:06.635 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.611+0000 7f274cb98640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f274810ce70 0x7f274819ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:25:06.635 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.611+0000 7f2747fff640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f27481038c0 0x7f27481985f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:45110/0 (socket says 172.21.15.142:45110) 2024-09-06T14:25:06.635 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.611+0000 7f2747fff640 1 -- 172.21.15.142:0/873758115 learned_addr learned my addr 172.21.15.142:0/873758115 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:25:06.635 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.611+0000 7f27477fe640 1 --2- 172.21.15.142:0/873758115 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f27481042c0 0x7f2748198b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:25:06.635 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.611+0000 7f274cb98640 1 -- 172.21.15.142:0/873758115 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f27481038c0 msgr2=0x7f27481985f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:25:06.635 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.611+0000 7f274cb98640 1 --2- 172.21.15.142:0/873758115 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f27481038c0 0x7f27481985f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:06.635 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.611+0000 7f274cb98640 1 -- 172.21.15.142:0/873758115 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f27481042c0 msgr2=0x7f2748198b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:25:06.635 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.611+0000 7f274cb98640 1 --2- 172.21.15.142:0/873758115 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f27481042c0 0x7f2748198b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:06.635 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.611+0000 7f274cb98640 1 -- 172.21.15.142:0/873758115 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f274819e6a0 con 0x7f274810ce70 2024-09-06T14:25:06.635 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.611+0000 7f274cb98640 1 --2- 172.21.15.142:0/873758115 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f274810ce70 0x7f274819ded0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f2740033040 tx=0x7f27400093d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:25:06.635 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.611+0000 7f27457fa640 1 -- 172.21.15.142:0/873758115 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2740003db0 con 0x7f274810ce70 2024-09-06T14:25:06.635 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.611+0000 7f274e5fb640 1 -- 172.21.15.142:0/873758115 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f274819e930 con 0x7f274810ce70 2024-09-06T14:25:06.635 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.611+0000 7f27457fa640 1 -- 172.21.15.142:0/873758115 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7f27400044c0 con 0x7f274810ce70 2024-09-06T14:25:06.636 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.611+0000 7f27457fa640 1 -- 172.21.15.142:0/873758115 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2740030b50 con 0x7f274810ce70 2024-09-06T14:25:06.636 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.611+0000 7f274e5fb640 1 -- 172.21.15.142:0/873758115 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2748101620 con 0x7f274810ce70 2024-09-06T14:25:06.636 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.612+0000 7f274e5fb640 1 -- 172.21.15.142:0/873758115 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2748103ca0 con 0x7f274810ce70 2024-09-06T14:25:06.636 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.613+0000 7f27457fa640 1 -- 172.21.15.142:0/873758115 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 22) ==== 99844+0+0 (secure 0 0 0) 0x7f2740030cf0 con 0x7f274810ce70 2024-09-06T14:25:06.636 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.613+0000 7f27457fa640 1 --2- 172.21.15.142:0/873758115 >> v2:172.21.15.142:6800/2944545612 conn(0x7f2734077620 0x7f2734079ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:25:06.636 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.614+0000 7f27457fa640 1 -- 172.21.15.142:0/873758115 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 6043+0+0 (secure 0 0 0) 0x7f27400bab60 con 0x7f274810ce70 2024-09-06T14:25:06.636 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.614+0000 7f2747fff640 1 --2- 172.21.15.142:0/873758115 >> v2:172.21.15.142:6800/2944545612 conn(0x7f2734077620 0x7f2734079ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:25:06.636 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.614+0000 7f2747fff640 1 --2- 172.21.15.142:0/873758115 >> v2:172.21.15.142:6800/2944545612 conn(0x7f2734077620 0x7f2734079ae0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f273800ac90 tx=0x7f273800a380 comp rx=0 tx=0).ready entity=mgr.14601 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:25:06.636 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.621+0000 7f27457fa640 1 -- 172.21.15.142:0/873758115 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7f2740087f00 con 0x7f274810ce70 2024-09-06T14:25:06.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.494+0000 7fa221093640 1 -- 172.21.15.142:0/2804280270 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa21c1073d0 msgr2=0x7fa21c107810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:25:06.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.494+0000 7fa221093640 1 --2- 172.21.15.142:0/2804280270 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa21c1073d0 0x7fa21c107810 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fa210009900 tx=0x7fa21002a410 comp rx=0 tx=0).stop 2024-09-06T14:25:06.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.494+0000 7fa221093640 1 -- 172.21.15.142:0/2804280270 shutdown_connections 2024-09-06T14:25:06.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.494+0000 7fa221093640 1 --2- 172.21.15.142:0/2804280270 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa21c107d50 0x7fa21c10e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:06.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.494+0000 7fa221093640 1 --2- 172.21.15.142:0/2804280270 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa21c1073d0 0x7fa21c107810 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:06.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.494+0000 7fa221093640 1 --2- 172.21.15.142:0/2804280270 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa21c1019c0 0x7fa21c101da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:06.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.494+0000 7fa221093640 1 -- 172.21.15.142:0/2804280270 >> 172.21.15.142:0/2804280270 conn(0x7fa21c0fd430 msgr2=0x7fa21c0ff850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:25:06.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.494+0000 7fa221093640 1 -- 172.21.15.142:0/2804280270 shutdown_connections 2024-09-06T14:25:06.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.494+0000 7fa221093640 1 -- 172.21.15.142:0/2804280270 wait complete. 2024-09-06T14:25:06.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.495+0000 7fa221093640 1 Processor -- start 2024-09-06T14:25:06.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.495+0000 7fa221093640 1 -- start start 2024-09-06T14:25:06.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.496+0000 7fa221093640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa21c1019c0 0x7fa21c19a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:25:06.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.496+0000 7fa221093640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa21c1073d0 0x7fa21c19ae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:25:06.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.496+0000 7fa221093640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa21c107d50 0x7fa21c1a0220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:25:06.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.496+0000 7fa221093640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fa21c111120 con 0x7fa21c1019c0 2024-09-06T14:25:06.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.496+0000 7fa221093640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa21c110fa0 con 0x7fa21c107d50 2024-09-06T14:25:06.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.496+0000 7fa221093640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7fa21c1112a0 con 0x7fa21c1073d0 2024-09-06T14:25:06.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.496+0000 7fa21a59c640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa21c1073d0 0x7fa21c19ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:25:06.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.496+0000 7fa21ad9d640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa21c1019c0 0x7fa21c19a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:25:06.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.496+0000 7fa21a59c640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa21c1073d0 0x7fa21c19ae80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:45100/0 (socket says 172.21.15.142:45100) 2024-09-06T14:25:06.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.496+0000 7fa21a59c640 1 -- 172.21.15.142:0/2920685708 learned_addr learned my addr 172.21.15.142:0/2920685708 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:25:06.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.496+0000 7fa21b59e640 1 --2- 172.21.15.142:0/2920685708 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa21c107d50 0x7fa21c1a0220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:25:06.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.496+0000 7fa21a59c640 1 -- 172.21.15.142:0/2920685708 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa21c107d50 msgr2=0x7fa21c1a0220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:25:06.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.496+0000 7fa21a59c640 1 --2- 172.21.15.142:0/2920685708 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa21c107d50 0x7fa21c1a0220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:06.744 INFO:teuthology.orchestra.run.smithi142.stdout:34359738415 2024-09-06T14:25:06.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.496+0000 7fa21a59c640 1 -- 172.21.15.142:0/2920685708 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa21c1019c0 msgr2=0x7fa21c19a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:25:06.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.496+0000 7fa21a59c640 1 --2- 172.21.15.142:0/2920685708 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa21c1019c0 0x7fa21c19a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:06.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.496+0000 7fa21a59c640 1 -- 172.21.15.142:0/2920685708 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa21c1a09f0 con 0x7fa21c1073d0 2024-09-06T14:25:06.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.496+0000 7fa21ad9d640 1 --2- 172.21.15.142:0/2920685708 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa21c1019c0 0x7fa21c19a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-06T14:25:06.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.496+0000 7fa21a59c640 1 --2- 172.21.15.142:0/2920685708 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa21c1073d0 0x7fa21c19ae80 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa210005940 tx=0x7fa210032c70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:25:06.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.496+0000 7fa203fff640 1 -- 172.21.15.142:0/2920685708 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa21003f030 con 0x7fa21c1073d0 2024-09-06T14:25:06.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.496+0000 7fa203fff640 1 -- 172.21.15.142:0/2920685708 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7fa210005480 con 0x7fa21c1073d0 2024-09-06T14:25:06.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.496+0000 7fa221093640 1 -- 172.21.15.142:0/2920685708 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa21c1a0c80 con 0x7fa21c1073d0 2024-09-06T14:25:06.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.496+0000 7fa203fff640 1 -- 172.21.15.142:0/2920685708 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa210035070 con 0x7fa21c1073d0 2024-09-06T14:25:06.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.496+0000 7fa221093640 1 -- 172.21.15.142:0/2920685708 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa21c1a73a0 con 0x7fa21c1073d0 2024-09-06T14:25:06.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.497+0000 7fa221093640 1 -- 172.21.15.142:0/2920685708 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa21c101da0 con 0x7fa21c1073d0 2024-09-06T14:25:06.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.498+0000 7fa203fff640 1 -- 172.21.15.142:0/2920685708 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 22) ==== 99844+0+0 (secure 0 0 0) 0x7fa210042030 con 0x7fa21c1073d0 2024-09-06T14:25:06.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.498+0000 7fa203fff640 1 --2- 172.21.15.142:0/2920685708 >> v2:172.21.15.142:6800/2944545612 conn(0x7fa204077620 0x7fa204079ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:25:06.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.499+0000 7fa21ad9d640 1 --2- 172.21.15.142:0/2920685708 >> v2:172.21.15.142:6800/2944545612 conn(0x7fa204077620 0x7fa204079ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:25:06.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.499+0000 7fa203fff640 1 -- 172.21.15.142:0/2920685708 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(62..62 src has 1..62) ==== 6043+0+0 (secure 0 0 0) 0x7fa2100b9e40 con 0x7fa21c1073d0 2024-09-06T14:25:06.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.499+0000 7fa21ad9d640 1 --2- 172.21.15.142:0/2920685708 >> v2:172.21.15.142:6800/2944545612 conn(0x7fa204077620 0x7fa204079ae0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fa20c009d60 tx=0x7fa20c009110 comp rx=0 tx=0).ready entity=mgr.14601 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:25:06.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.502+0000 7fa203fff640 1 -- 172.21.15.142:0/2920685708 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7fa210082f60 con 0x7fa21c1073d0 2024-09-06T14:25:06.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.669+0000 7fa221093640 1 -- 172.21.15.142:0/2920685708 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fa21c1a7570 con 0x7fa21c1073d0 2024-09-06T14:25:06.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.669+0000 7fa203fff640 1 -- 172.21.15.142:0/2920685708 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fa2100870d0 con 0x7fa21c1073d0 2024-09-06T14:25:06.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.671+0000 7fa221093640 1 -- 172.21.15.142:0/2920685708 >> v2:172.21.15.142:6800/2944545612 conn(0x7fa204077620 msgr2=0x7fa204079ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:25:06.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.671+0000 7fa221093640 1 --2- 172.21.15.142:0/2920685708 >> v2:172.21.15.142:6800/2944545612 conn(0x7fa204077620 0x7fa204079ae0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fa20c009d60 tx=0x7fa20c009110 comp rx=0 tx=0).stop 2024-09-06T14:25:06.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.671+0000 7fa221093640 1 -- 172.21.15.142:0/2920685708 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa21c1073d0 msgr2=0x7fa21c19ae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:25:06.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.671+0000 7fa221093640 1 --2- 172.21.15.142:0/2920685708 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa21c1073d0 0x7fa21c19ae80 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa210005940 tx=0x7fa210032c70 comp rx=0 tx=0).stop 2024-09-06T14:25:06.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.671+0000 7fa221093640 1 -- 172.21.15.142:0/2920685708 shutdown_connections 2024-09-06T14:25:06.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.672+0000 7fa221093640 1 --2- 172.21.15.142:0/2920685708 >> v2:172.21.15.142:6800/2944545612 conn(0x7fa204077620 0x7fa204079ae0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:06.747 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.672+0000 7fa221093640 1 --2- 172.21.15.142:0/2920685708 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa21c107d50 0x7fa21c1a0220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:06.747 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.672+0000 7fa221093640 1 --2- 172.21.15.142:0/2920685708 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa21c1073d0 0x7fa21c19ae80 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:06.747 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.672+0000 7fa221093640 1 --2- 172.21.15.142:0/2920685708 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa21c1019c0 0x7fa21c19a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:06.747 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.672+0000 7fa221093640 1 -- 172.21.15.142:0/2920685708 >> 172.21.15.142:0/2920685708 conn(0x7fa21c0fd430 msgr2=0x7fa21c06b000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:25:06.747 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.672+0000 7fa221093640 1 -- 172.21.15.142:0/2920685708 shutdown_connections 2024-09-06T14:25:06.747 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.672+0000 7fa221093640 1 -- 172.21.15.142:0/2920685708 wait complete. 2024-09-06T14:25:06.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.785+0000 7f274e5fb640 1 -- 172.21.15.142:0/873758115 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f2748107ee0 con 0x7f274810ce70 2024-09-06T14:25:06.782 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.786+0000 7f27457fa640 1 -- 172.21.15.142:0/873758115 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f274008c240 con 0x7f274810ce70 2024-09-06T14:25:06.783 INFO:teuthology.orchestra.run.smithi142.stdout:55834574889 2024-09-06T14:25:06.785 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.788+0000 7f274e5fb640 1 -- 172.21.15.142:0/873758115 >> v2:172.21.15.142:6800/2944545612 conn(0x7f2734077620 msgr2=0x7f2734079ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:25:06.785 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.788+0000 7f274e5fb640 1 --2- 172.21.15.142:0/873758115 >> v2:172.21.15.142:6800/2944545612 conn(0x7f2734077620 0x7f2734079ae0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f273800ac90 tx=0x7f273800a380 comp rx=0 tx=0).stop 2024-09-06T14:25:06.785 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.788+0000 7f274e5fb640 1 -- 172.21.15.142:0/873758115 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f274810ce70 msgr2=0x7f274819ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:25:06.785 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.788+0000 7f274e5fb640 1 --2- 172.21.15.142:0/873758115 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f274810ce70 0x7f274819ded0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f2740033040 tx=0x7f27400093d0 comp rx=0 tx=0).stop 2024-09-06T14:25:06.785 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.788+0000 7f274e5fb640 1 -- 172.21.15.142:0/873758115 shutdown_connections 2024-09-06T14:25:06.785 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.788+0000 7f274e5fb640 1 --2- 172.21.15.142:0/873758115 >> v2:172.21.15.142:6800/2944545612 conn(0x7f2734077620 0x7f2734079ae0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:06.785 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.788+0000 7f274e5fb640 1 --2- 172.21.15.142:0/873758115 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f274810ce70 0x7f274819ded0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:06.785 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.788+0000 7f274e5fb640 1 --2- 172.21.15.142:0/873758115 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f27481042c0 0x7f2748198b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:06.785 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.788+0000 7f274e5fb640 1 --2- 172.21.15.142:0/873758115 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f27481038c0 0x7f27481985f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:06.785 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.788+0000 7f274e5fb640 1 -- 172.21.15.142:0/873758115 >> 172.21.15.142:0/873758115 conn(0x7f27480fd430 msgr2=0x7f27480fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:25:06.786 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.788+0000 7f274e5fb640 1 -- 172.21.15.142:0/873758115 shutdown_connections 2024-09-06T14:25:06.786 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:06.788+0000 7f274e5fb640 1 -- 172.21.15.142:0/873758115 wait complete. 2024-09-06T14:25:07.024 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:06 smithi142 ceph-mon[37372]: pgmap v8: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-06T14:25:07.024 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:06 smithi142 ceph-mon[37372]: from='client.? 172.21.15.142:0/2920685708' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-06T14:25:07.025 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:06 smithi142 ceph-mon[29958]: pgmap v8: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-06T14:25:07.025 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:06 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/2920685708' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-06T14:25:07.047 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 06 14:25:07 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-iscsi-iscsi-a[59853]: debug there is no tcmu-runner data available 2024-09-06T14:25:07.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:06 smithi162 ceph-mon[33787]: pgmap v8: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-06T14:25:07.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:06 smithi162 ceph-mon[33787]: from='client.? 172.21.15.142:0/2920685708' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-06T14:25:07.714 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738414 got 34359738415 for osd.0 2024-09-06T14:25:07.714 DEBUG:teuthology.parallel:result is None 2024-09-06T14:25:07.958 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:07 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/873758115' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-06T14:25:07.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:07 smithi142 ceph-mon[29958]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-06T14:25:07.959 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:07 smithi142 ceph-mon[37372]: from='client.? 172.21.15.142:0/873758115' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-06T14:25:07.959 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:07 smithi142 ceph-mon[37372]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-06T14:25:08.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:07 smithi162 ceph-mon[33787]: from='client.? 172.21.15.142:0/873758115' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-06T14:25:08.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:07 smithi162 ceph-mon[33787]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-06T14:25:08.410 INFO:tasks.cephadm.ceph_manager.ceph:need seq 180388626446 got 180388626448 for osd.6 2024-09-06T14:25:08.411 DEBUG:teuthology.parallel:result is None 2024-09-06T14:25:08.838 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574888 got 55834574889 for osd.1 2024-09-06T14:25:08.838 DEBUG:teuthology.parallel:result is None 2024-09-06T14:25:09.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:08 smithi162 ceph-mon[33787]: pgmap v9: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-06T14:25:09.098 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:08 smithi142 ceph-mon[29958]: pgmap v9: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-06T14:25:09.099 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:08 smithi142 ceph-mon[37372]: pgmap v9: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-06T14:25:09.532 INFO:tasks.cephadm.ceph_manager.ceph:need seq 206158430218 got 206158430219 for osd.7 2024-09-06T14:25:09.533 DEBUG:teuthology.parallel:result is None 2024-09-06T14:25:09.799 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:25:10.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:10.322+0000 7fc50e24a640 1 -- 172.21.15.142:0/3892738828 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc508108e70 msgr2=0x7fc50810f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:25:10.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:10.322+0000 7fc50e24a640 1 --2- 172.21.15.142:0/3892738828 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc508108e70 0x7fc50810f670 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fc500009860 tx=0x7fc50002e750 comp rx=0 tx=0).stop 2024-09-06T14:25:10.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:10.322+0000 7fc50e24a640 1 -- 172.21.15.142:0/3892738828 shutdown_connections 2024-09-06T14:25:10.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:10.322+0000 7fc50e24a640 1 --2- 172.21.15.142:0/3892738828 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc508108e70 0x7fc50810f670 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:10.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:10.322+0000 7fc50e24a640 1 --2- 172.21.15.142:0/3892738828 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc5081084f0 0x7fc508108930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:10.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:10.322+0000 7fc50e24a640 1 --2- 172.21.15.142:0/3892738828 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc508102ae0 0x7fc508102ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:10.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:10.322+0000 7fc50e24a640 1 -- 172.21.15.142:0/3892738828 >> 172.21.15.142:0/3892738828 conn(0x7fc5080fe530 msgr2=0x7fc508100950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:25:10.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:10.322+0000 7fc50e24a640 1 -- 172.21.15.142:0/3892738828 shutdown_connections 2024-09-06T14:25:10.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:10.322+0000 7fc50e24a640 1 -- 172.21.15.142:0/3892738828 wait complete. 2024-09-06T14:25:10.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:10.323+0000 7fc50e24a640 1 Processor -- start 2024-09-06T14:25:10.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:10.323+0000 7fc50e24a640 1 -- start start 2024-09-06T14:25:10.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:10.323+0000 7fc50e24a640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc508102ae0 0x7fc508196470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:25:10.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:10.323+0000 7fc50e24a640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc5081084f0 0x7fc5081969b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:25:10.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:10.323+0000 7fc50e24a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc508108e70 0x7fc50819bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:25:10.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:10.323+0000 7fc50e24a640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fc508077080 con 0x7fc5081084f0 2024-09-06T14:25:10.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:10.323+0000 7fc50e24a640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc508076f00 con 0x7fc508108e70 2024-09-06T14:25:10.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:10.323+0000 7fc50e24a640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7fc5081102f0 con 0x7fc508102ae0 2024-09-06T14:25:10.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:10.323+0000 7fc5077fe640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc508102ae0 0x7fc508196470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:25:10.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:10.323+0000 7fc5077fe640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc508102ae0 0x7fc508196470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:35640/0 (socket says 172.21.15.142:35640) 2024-09-06T14:25:10.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:10.323+0000 7fc5077fe640 1 -- 172.21.15.142:0/134902297 learned_addr learned my addr 172.21.15.142:0/134902297 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:25:10.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:10.324+0000 7fc506ffd640 1 --2- 172.21.15.142:0/134902297 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc5081084f0 0x7fc5081969b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:25:10.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:10.324+0000 7fc507fff640 1 --2- 172.21.15.142:0/134902297 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc508108e70 0x7fc50819bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:25:10.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:10.324+0000 7fc5077fe640 1 -- 172.21.15.142:0/134902297 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc508108e70 msgr2=0x7fc50819bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:25:10.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:10.324+0000 7fc5077fe640 1 --2- 172.21.15.142:0/134902297 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc508108e70 0x7fc50819bd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:10.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:10.324+0000 7fc5077fe640 1 -- 172.21.15.142:0/134902297 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc5081084f0 msgr2=0x7fc5081969b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:25:10.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:10.324+0000 7fc5077fe640 1 --2- 172.21.15.142:0/134902297 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc5081084f0 0x7fc5081969b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:10.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:10.324+0000 7fc5077fe640 1 -- 172.21.15.142:0/134902297 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc50819c520 con 0x7fc508102ae0 2024-09-06T14:25:10.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:10.324+0000 7fc506ffd640 1 --2- 172.21.15.142:0/134902297 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc5081084f0 0x7fc5081969b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-06T14:25:10.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:10.324+0000 7fc507fff640 1 --2- 172.21.15.142:0/134902297 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc508108e70 0x7fc50819bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-06T14:25:10.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:10.324+0000 7fc5077fe640 1 --2- 172.21.15.142:0/134902297 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc508102ae0 0x7fc508196470 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc4f800c420 tx=0x7fc4f8009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:25:10.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:10.324+0000 7fc504ff9640 1 -- 172.21.15.142:0/134902297 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4f8003970 con 0x7fc508102ae0 2024-09-06T14:25:10.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:10.324+0000 7fc504ff9640 1 -- 172.21.15.142:0/134902297 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7fc4f8004020 con 0x7fc508102ae0 2024-09-06T14:25:10.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:10.324+0000 7fc50e24a640 1 -- 172.21.15.142:0/134902297 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc50819c810 con 0x7fc508102ae0 2024-09-06T14:25:10.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:10.324+0000 7fc504ff9640 1 -- 172.21.15.142:0/134902297 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4f800b8b0 con 0x7fc508102ae0 2024-09-06T14:25:10.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:10.324+0000 7fc50e24a640 1 -- 172.21.15.142:0/134902297 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5081a2ed0 con 0x7fc508102ae0 2024-09-06T14:25:10.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:10.325+0000 7fc50e24a640 1 -- 172.21.15.142:0/134902297 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc508102ec0 con 0x7fc508102ae0 2024-09-06T14:25:10.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:10.326+0000 7fc504ff9640 1 -- 172.21.15.142:0/134902297 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 22) ==== 99844+0+0 (secure 0 0 0) 0x7fc4f801d020 con 0x7fc508102ae0 2024-09-06T14:25:10.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:10.326+0000 7fc504ff9640 1 --2- 172.21.15.142:0/134902297 >> v2:172.21.15.142:6800/2944545612 conn(0x7fc4f0077620 0x7fc4f0079ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:25:10.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:10.326+0000 7fc506ffd640 1 --2- 172.21.15.142:0/134902297 >> v2:172.21.15.142:6800/2944545612 conn(0x7fc4f0077620 0x7fc4f0079ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:25:10.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:10.326+0000 7fc504ff9640 1 -- 172.21.15.142:0/134902297 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(62..62 src has 1..62) ==== 6043+0+0 (secure 0 0 0) 0x7fc4f80998a0 con 0x7fc508102ae0 2024-09-06T14:25:10.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:10.327+0000 7fc506ffd640 1 --2- 172.21.15.142:0/134902297 >> v2:172.21.15.142:6800/2944545612 conn(0x7fc4f0077620 0x7fc4f0079ae0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fc508100e60 tx=0x7fc4fc00b390 comp rx=0 tx=0).ready entity=mgr.14601 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:25:10.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:10.330+0000 7fc504ff9640 1 -- 172.21.15.142:0/134902297 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7fc4f80629c0 con 0x7fc508102ae0 2024-09-06T14:25:10.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:10.503+0000 7fc50e24a640 1 -- 172.21.15.142:0/134902297 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fc5081a3100 con 0x7fc508102ae0 2024-09-06T14:25:10.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:10.504+0000 7fc504ff9640 1 -- 172.21.15.142:0/134902297 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fc4f8069e20 con 0x7fc508102ae0 2024-09-06T14:25:10.501 INFO:teuthology.orchestra.run.smithi142.stdout:154618822678 2024-09-06T14:25:10.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:10.506+0000 7fc50e24a640 1 -- 172.21.15.142:0/134902297 >> v2:172.21.15.142:6800/2944545612 conn(0x7fc4f0077620 msgr2=0x7fc4f0079ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:25:10.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:10.506+0000 7fc50e24a640 1 --2- 172.21.15.142:0/134902297 >> v2:172.21.15.142:6800/2944545612 conn(0x7fc4f0077620 0x7fc4f0079ae0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fc508100e60 tx=0x7fc4fc00b390 comp rx=0 tx=0).stop 2024-09-06T14:25:10.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:10.506+0000 7fc50e24a640 1 -- 172.21.15.142:0/134902297 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc508102ae0 msgr2=0x7fc508196470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:25:10.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:10.506+0000 7fc50e24a640 1 --2- 172.21.15.142:0/134902297 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc508102ae0 0x7fc508196470 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc4f800c420 tx=0x7fc4f8009f90 comp rx=0 tx=0).stop 2024-09-06T14:25:10.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:10.507+0000 7fc50e24a640 1 -- 172.21.15.142:0/134902297 shutdown_connections 2024-09-06T14:25:10.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:10.507+0000 7fc50e24a640 1 --2- 172.21.15.142:0/134902297 >> v2:172.21.15.142:6800/2944545612 conn(0x7fc4f0077620 0x7fc4f0079ae0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:10.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:10.507+0000 7fc50e24a640 1 --2- 172.21.15.142:0/134902297 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc508108e70 0x7fc50819bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:10.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:10.507+0000 7fc50e24a640 1 --2- 172.21.15.142:0/134902297 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc5081084f0 0x7fc5081969b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:10.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:10.507+0000 7fc50e24a640 1 --2- 172.21.15.142:0/134902297 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc508102ae0 0x7fc508196470 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:10.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:10.507+0000 7fc50e24a640 1 -- 172.21.15.142:0/134902297 >> 172.21.15.142:0/134902297 conn(0x7fc5080fe530 msgr2=0x7fc5080fecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:25:10.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:10.507+0000 7fc50e24a640 1 -- 172.21.15.142:0/134902297 shutdown_connections 2024-09-06T14:25:10.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:10.507+0000 7fc50e24a640 1 -- 172.21.15.142:0/134902297 wait complete. 2024-09-06T14:25:11.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:10 smithi162 ceph-mon[33787]: pgmap v10: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-06T14:25:11.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:10 smithi162 ceph-mon[33787]: from='client.? 172.21.15.142:0/134902297' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-06T14:25:11.091 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:10 smithi142 ceph-mon[29958]: pgmap v10: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-06T14:25:11.091 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:10 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/134902297' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-06T14:25:11.091 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:10 smithi142 ceph-mon[37372]: pgmap v10: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-06T14:25:11.091 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:10 smithi142 ceph-mon[37372]: from='client.? 172.21.15.142:0/134902297' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-06T14:25:11.128 INFO:tasks.cephadm.ceph_manager.ceph:need seq 154618822676 got 154618822678 for osd.5 2024-09-06T14:25:11.128 DEBUG:teuthology.parallel:result is None 2024-09-06T14:25:11.128 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-06T14:25:11.129 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph pg dump --format=json 2024-09-06T14:25:11.392 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:25:11.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:11.910+0000 7fbd3ce73640 1 -- 172.21.15.142:0/3025174581 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbd381019c0 msgr2=0x7fbd38101da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:25:11.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:11.910+0000 7fbd3ce73640 1 --2- 172.21.15.142:0/3025174581 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbd381019c0 0x7fbd38101da0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fbd28009900 tx=0x7fbd2802a410 comp rx=0 tx=0).stop 2024-09-06T14:25:11.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:11.910+0000 7fbd3ce73640 1 -- 172.21.15.142:0/3025174581 shutdown_connections 2024-09-06T14:25:11.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:11.910+0000 7fbd3ce73640 1 --2- 172.21.15.142:0/3025174581 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbd38107d50 0x7fbd3810e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:11.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:11.910+0000 7fbd3ce73640 1 --2- 172.21.15.142:0/3025174581 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbd381073d0 0x7fbd38107810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:11.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:11.910+0000 7fbd3ce73640 1 --2- 172.21.15.142:0/3025174581 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbd381019c0 0x7fbd38101da0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:11.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:11.910+0000 7fbd3ce73640 1 -- 172.21.15.142:0/3025174581 >> 172.21.15.142:0/3025174581 conn(0x7fbd380fd430 msgr2=0x7fbd380ff850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:25:11.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:11.910+0000 7fbd3ce73640 1 -- 172.21.15.142:0/3025174581 shutdown_connections 2024-09-06T14:25:11.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:11.910+0000 7fbd3ce73640 1 -- 172.21.15.142:0/3025174581 wait complete. 2024-09-06T14:25:11.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:11.911+0000 7fbd3ce73640 1 Processor -- start 2024-09-06T14:25:11.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:11.911+0000 7fbd3ce73640 1 -- start start 2024-09-06T14:25:11.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:11.911+0000 7fbd3ce73640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbd381019c0 0x7fbd38198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:25:11.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:11.911+0000 7fbd3ce73640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbd381073d0 0x7fbd38198c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:25:11.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:11.911+0000 7fbd3ce73640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbd38107d50 0x7fbd3819dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:25:11.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:11.911+0000 7fbd3ce73640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fbd38112220 con 0x7fbd381019c0 2024-09-06T14:25:11.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:11.911+0000 7fbd3ce73640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fbd381120a0 con 0x7fbd38107d50 2024-09-06T14:25:11.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:11.911+0000 7fbd3ce73640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7fbd381123a0 con 0x7fbd381073d0 2024-09-06T14:25:11.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:11.911+0000 7fbd35d9b640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbd381073d0 0x7fbd38198c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:25:11.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:11.911+0000 7fbd3659c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbd381019c0 0x7fbd38198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:25:11.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:11.911+0000 7fbd35d9b640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbd381073d0 0x7fbd38198c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:35662/0 (socket says 172.21.15.142:35662) 2024-09-06T14:25:11.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:11.911+0000 7fbd3659c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbd381019c0 0x7fbd38198710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:58252/0 (socket says 172.21.15.142:58252) 2024-09-06T14:25:11.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:11.911+0000 7fbd35d9b640 1 -- 172.21.15.142:0/846656465 learned_addr learned my addr 172.21.15.142:0/846656465 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:25:11.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:11.911+0000 7fbd36d9d640 1 --2- 172.21.15.142:0/846656465 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbd38107d50 0x7fbd3819dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:25:11.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:11.912+0000 7fbd3659c640 1 -- 172.21.15.142:0/846656465 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbd381073d0 msgr2=0x7fbd38198c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:25:11.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:11.912+0000 7fbd3659c640 1 --2- 172.21.15.142:0/846656465 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbd381073d0 0x7fbd38198c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:11.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:11.912+0000 7fbd3659c640 1 -- 172.21.15.142:0/846656465 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbd38107d50 msgr2=0x7fbd3819dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:25:11.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:11.912+0000 7fbd3659c640 1 --2- 172.21.15.142:0/846656465 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbd38107d50 0x7fbd3819dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:11.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:11.912+0000 7fbd3659c640 1 -- 172.21.15.142:0/846656465 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd3819e7c0 con 0x7fbd381019c0 2024-09-06T14:25:11.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:11.912+0000 7fbd3659c640 1 --2- 172.21.15.142:0/846656465 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbd381019c0 0x7fbd38198710 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fbd2802a930 tx=0x7fbd280329c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:25:11.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:11.912+0000 7fbd277fe640 1 -- 172.21.15.142:0/846656465 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd2803f030 con 0x7fbd381019c0 2024-09-06T14:25:11.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:11.912+0000 7fbd3ce73640 1 -- 172.21.15.142:0/846656465 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd3819ea50 con 0x7fbd381019c0 2024-09-06T14:25:11.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:11.912+0000 7fbd277fe640 1 -- 172.21.15.142:0/846656465 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7fbd280044c0 con 0x7fbd381019c0 2024-09-06T14:25:11.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:11.912+0000 7fbd277fe640 1 -- 172.21.15.142:0/846656465 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd28035070 con 0x7fbd381019c0 2024-09-06T14:25:11.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:11.912+0000 7fbd3ce73640 1 -- 172.21.15.142:0/846656465 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd381a5170 con 0x7fbd381019c0 2024-09-06T14:25:11.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:11.913+0000 7fbd3ce73640 1 -- 172.21.15.142:0/846656465 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd38101da0 con 0x7fbd381019c0 2024-09-06T14:25:11.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:11.914+0000 7fbd277fe640 1 -- 172.21.15.142:0/846656465 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 22) ==== 99844+0+0 (secure 0 0 0) 0x7fbd28042030 con 0x7fbd381019c0 2024-09-06T14:25:11.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:11.914+0000 7fbd277fe640 1 --2- 172.21.15.142:0/846656465 >> v2:172.21.15.142:6800/2944545612 conn(0x7fbd20077620 0x7fbd20079ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:25:11.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:11.914+0000 7fbd35d9b640 1 --2- 172.21.15.142:0/846656465 >> v2:172.21.15.142:6800/2944545612 conn(0x7fbd20077620 0x7fbd20079ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:25:11.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:11.914+0000 7fbd277fe640 1 -- 172.21.15.142:0/846656465 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 6043+0+0 (secure 0 0 0) 0x7fbd280b9d80 con 0x7fbd381019c0 2024-09-06T14:25:11.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:11.915+0000 7fbd35d9b640 1 --2- 172.21.15.142:0/846656465 >> v2:172.21.15.142:6800/2944545612 conn(0x7fbd20077620 0x7fbd20079ae0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fbd2c0095f0 tx=0x7fbd2c005f90 comp rx=0 tx=0).ready entity=mgr.14601 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:25:11.915 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:11.918+0000 7fbd277fe640 1 -- 172.21.15.142:0/846656465 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7fbd28087010 con 0x7fbd381019c0 2024-09-06T14:25:12.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:11 smithi162 ceph-mon[33787]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-06T14:25:12.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:11 smithi162 ceph-mon[33787]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:12.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:11 smithi162 ceph-mon[33787]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:12.078 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:12.081+0000 7fbd3ce73640 1 -- 172.21.15.142:0/846656465 --> v2:172.21.15.142:6800/2944545612 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fbd380632f0 con 0x7fbd20077620 2024-09-06T14:25:12.078 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:11 smithi142 ceph-mon[29958]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-06T14:25:12.078 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:11 smithi142 ceph-mon[29958]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:12.078 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:11 smithi142 ceph-mon[29958]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:12.078 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:11 smithi142 ceph-mon[37372]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-06T14:25:12.079 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:11 smithi142 ceph-mon[37372]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:12.079 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:11 smithi142 ceph-mon[37372]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:12.084 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:12.087+0000 7fbd277fe640 1 -- 172.21.15.142:0/846656465 <== mgr.14601 v2:172.21.15.142:6800/2944545612 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+346465 (secure 0 0 0) 0x7fbd380632f0 con 0x7fbd20077620 2024-09-06T14:25:12.087 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:25:12.089 INFO:teuthology.orchestra.run.smithi142.stderr:dumped all 2024-09-06T14:25:12.092 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:12.090+0000 7fbd3ce73640 1 -- 172.21.15.142:0/846656465 >> v2:172.21.15.142:6800/2944545612 conn(0x7fbd20077620 msgr2=0x7fbd20079ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:25:12.092 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:12.090+0000 7fbd3ce73640 1 --2- 172.21.15.142:0/846656465 >> v2:172.21.15.142:6800/2944545612 conn(0x7fbd20077620 0x7fbd20079ae0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fbd2c0095f0 tx=0x7fbd2c005f90 comp rx=0 tx=0).stop 2024-09-06T14:25:12.092 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:12.090+0000 7fbd3ce73640 1 -- 172.21.15.142:0/846656465 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbd381019c0 msgr2=0x7fbd38198710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:25:12.092 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:12.090+0000 7fbd3ce73640 1 --2- 172.21.15.142:0/846656465 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbd381019c0 0x7fbd38198710 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fbd2802a930 tx=0x7fbd280329c0 comp rx=0 tx=0).stop 2024-09-06T14:25:12.092 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:12.091+0000 7fbd3ce73640 1 -- 172.21.15.142:0/846656465 shutdown_connections 2024-09-06T14:25:12.092 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:12.091+0000 7fbd3ce73640 1 --2- 172.21.15.142:0/846656465 >> v2:172.21.15.142:6800/2944545612 conn(0x7fbd20077620 0x7fbd20079ae0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:12.092 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:12.091+0000 7fbd3ce73640 1 --2- 172.21.15.142:0/846656465 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbd38107d50 0x7fbd3819dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:12.092 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:12.091+0000 7fbd3ce73640 1 --2- 172.21.15.142:0/846656465 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbd381073d0 0x7fbd38198c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:12.092 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:12.091+0000 7fbd3ce73640 1 --2- 172.21.15.142:0/846656465 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbd381019c0 0x7fbd38198710 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:12.093 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:12.091+0000 7fbd3ce73640 1 -- 172.21.15.142:0/846656465 >> 172.21.15.142:0/846656465 conn(0x7fbd380fd430 msgr2=0x7fbd380ff820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:25:12.093 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:12.091+0000 7fbd3ce73640 1 -- 172.21.15.142:0/846656465 shutdown_connections 2024-09-06T14:25:12.093 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:12.091+0000 7fbd3ce73640 1 -- 172.21.15.142:0/846656465 wait complete. 2024-09-06T14:25:12.956 INFO:teuthology.orchestra.run.smithi142.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2024-09-06T14:25:11.517482+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":888,"num_read_kb":745,"num_write":549,"num_write_kb":1673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":552,"ondisk_log_size":552,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":239540,"kb_used_data":6684,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":749721676,"statfs":{"total":767960285184,"available":767714996224,"internally_reserved":0,"allocated":6844416,"data_stored":3633129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":75,"num_read_kb":191,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003140"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":30,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.449767+0000","last_change":"2024-09-06T14:24:25.997015+0000","last_active":"2024-09-06T14:24:55.449767+0000","last_peered":"2024-09-06T14:24:55.449767+0000","last_clean":"2024-09-06T14:24:55.449767+0000","last_became_active":"2024-09-06T14:24:25.996213+0000","last_became_peered":"2024-09-06T14:24:25.996213+0000","last_unstale":"2024-09-06T14:24:55.449767+0000","last_undegraded":"2024-09-06T14:24:55.449767+0000","last_fullsized":"2024-09-06T14:24:55.449767+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_clean_scrub_stamp":"2024-09-06T14:24:24.970251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T20:14:30.434781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"60'9","reported_seq":42,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.921464+0000","last_change":"2024-09-06T14:24:28.006668+0000","last_active":"2024-09-06T14:24:55.921464+0000","last_peered":"2024-09-06T14:24:55.921464+0000","last_clean":"2024-09-06T14:24:55.921464+0000","last_became_active":"2024-09-06T14:24:28.006320+0000","last_became_peered":"2024-09-06T14:24:28.006320+0000","last_unstale":"2024-09-06T14:24:55.921464+0000","last_undegraded":"2024-09-06T14:24:55.921464+0000","last_fullsized":"2024-09-06T14:24:55.921464+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_clean_scrub_stamp":"2024-09-06T14:24:26.984416+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T19:18:48.111200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":22,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.922514+0000","last_change":"2024-09-06T14:24:30.018374+0000","last_active":"2024-09-06T14:24:55.922514+0000","last_peered":"2024-09-06T14:24:55.922514+0000","last_clean":"2024-09-06T14:24:55.922514+0000","last_became_active":"2024-09-06T14:24:30.018083+0000","last_became_peered":"2024-09-06T14:24:30.018083+0000","last_unstale":"2024-09-06T14:24:55.922514+0000","last_undegraded":"2024-09-06T14:24:55.922514+0000","last_fullsized":"2024-09-06T14:24:55.922514+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_clean_scrub_stamp":"2024-09-06T14:24:28.992633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T21:32:36.013547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":18,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.921684+0000","last_change":"2024-09-06T14:24:32.031400+0000","last_active":"2024-09-06T14:24:55.921684+0000","last_peered":"2024-09-06T14:24:55.921684+0000","last_clean":"2024-09-06T14:24:55.921684+0000","last_became_active":"2024-09-06T14:24:32.031142+0000","last_became_peered":"2024-09-06T14:24:32.031142+0000","last_unstale":"2024-09-06T14:24:55.921684+0000","last_undegraded":"2024-09-06T14:24:55.921684+0000","last_fullsized":"2024-09-06T14:24:55.921684+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_clean_scrub_stamp":"2024-09-06T14:24:31.009248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T15:39:27.650963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":18,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.449715+0000","last_change":"2024-09-06T14:24:32.038789+0000","last_active":"2024-09-06T14:24:55.449715+0000","last_peered":"2024-09-06T14:24:55.449715+0000","last_clean":"2024-09-06T14:24:55.449715+0000","last_became_active":"2024-09-06T14:24:32.038081+0000","last_became_peered":"2024-09-06T14:24:32.038081+0000","last_unstale":"2024-09-06T14:24:55.449715+0000","last_undegraded":"2024-09-06T14:24:55.449715+0000","last_fullsized":"2024-09-06T14:24:55.449715+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_clean_scrub_stamp":"2024-09-06T14:24:31.009248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T15:01:32.703233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":30,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.451483+0000","last_change":"2024-09-06T14:24:25.999476+0000","last_active":"2024-09-06T14:24:55.451483+0000","last_peered":"2024-09-06T14:24:55.451483+0000","last_clean":"2024-09-06T14:24:55.451483+0000","last_became_active":"2024-09-06T14:24:25.998488+0000","last_became_peered":"2024-09-06T14:24:25.998488+0000","last_unstale":"2024-09-06T14:24:55.451483+0000","last_undegraded":"2024-09-06T14:24:55.451483+0000","last_fullsized":"2024-09-06T14:24:55.451483+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_clean_scrub_stamp":"2024-09-06T14:24:24.970251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T23:24:57.206743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"60'15","reported_seq":51,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.450056+0000","last_change":"2024-09-06T14:24:28.014521+0000","last_active":"2024-09-06T14:24:55.450056+0000","last_peered":"2024-09-06T14:24:55.450056+0000","last_clean":"2024-09-06T14:24:55.450056+0000","last_became_active":"2024-09-06T14:24:28.014236+0000","last_became_peered":"2024-09-06T14:24:28.014236+0000","last_unstale":"2024-09-06T14:24:55.450056+0000","last_undegraded":"2024-09-06T14:24:55.450056+0000","last_fullsized":"2024-09-06T14:24:55.450056+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_clean_scrub_stamp":"2024-09-06T14:24:26.984416+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T19:33:16.100900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":22,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.921457+0000","last_change":"2024-09-06T14:24:30.017693+0000","last_active":"2024-09-06T14:24:55.921457+0000","last_peered":"2024-09-06T14:24:55.921457+0000","last_clean":"2024-09-06T14:24:55.921457+0000","last_became_active":"2024-09-06T14:24:30.017390+0000","last_became_peered":"2024-09-06T14:24:30.017390+0000","last_unstale":"2024-09-06T14:24:55.921457+0000","last_undegraded":"2024-09-06T14:24:55.921457+0000","last_fullsized":"2024-09-06T14:24:55.921457+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_clean_scrub_stamp":"2024-09-06T14:24:28.992633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T22:12:26.568897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":30,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.786297+0000","last_change":"2024-09-06T14:24:25.996508+0000","last_active":"2024-09-06T14:24:55.786297+0000","last_peered":"2024-09-06T14:24:55.786297+0000","last_clean":"2024-09-06T14:24:55.786297+0000","last_became_active":"2024-09-06T14:24:25.996217+0000","last_became_peered":"2024-09-06T14:24:25.996217+0000","last_unstale":"2024-09-06T14:24:55.786297+0000","last_undegraded":"2024-09-06T14:24:55.786297+0000","last_fullsized":"2024-09-06T14:24:55.786297+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_clean_scrub_stamp":"2024-09-06T14:24:24.970251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-08T01:15:20.055475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"60'9","reported_seq":42,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.922822+0000","last_change":"2024-09-06T14:24:28.012532+0000","last_active":"2024-09-06T14:24:55.922822+0000","last_peered":"2024-09-06T14:24:55.922822+0000","last_clean":"2024-09-06T14:24:55.922822+0000","last_became_active":"2024-09-06T14:24:28.012241+0000","last_became_peered":"2024-09-06T14:24:28.012241+0000","last_unstale":"2024-09-06T14:24:55.922822+0000","last_undegraded":"2024-09-06T14:24:55.922822+0000","last_fullsized":"2024-09-06T14:24:55.922822+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_clean_scrub_stamp":"2024-09-06T14:24:26.984416+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T15:00:50.407496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":22,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.785848+0000","last_change":"2024-09-06T14:24:30.016494+0000","last_active":"2024-09-06T14:24:55.785848+0000","last_peered":"2024-09-06T14:24:55.785848+0000","last_clean":"2024-09-06T14:24:55.785848+0000","last_became_active":"2024-09-06T14:24:30.015964+0000","last_became_peered":"2024-09-06T14:24:30.015964+0000","last_unstale":"2024-09-06T14:24:55.785848+0000","last_undegraded":"2024-09-06T14:24:55.785848+0000","last_fullsized":"2024-09-06T14:24:55.785848+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_clean_scrub_stamp":"2024-09-06T14:24:28.992633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-08T00:07:54.412112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":18,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.450243+0000","last_change":"2024-09-06T14:24:32.034268+0000","last_active":"2024-09-06T14:24:55.450243+0000","last_peered":"2024-09-06T14:24:55.450243+0000","last_clean":"2024-09-06T14:24:55.450243+0000","last_became_active":"2024-09-06T14:24:32.033955+0000","last_became_peered":"2024-09-06T14:24:32.033955+0000","last_unstale":"2024-09-06T14:24:55.450243+0000","last_undegraded":"2024-09-06T14:24:55.450243+0000","last_fullsized":"2024-09-06T14:24:55.450243+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_clean_scrub_stamp":"2024-09-06T14:24:31.009248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-08T00:37:48.342424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":30,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.786032+0000","last_change":"2024-09-06T14:24:25.997921+0000","last_active":"2024-09-06T14:24:55.786032+0000","last_peered":"2024-09-06T14:24:55.786032+0000","last_clean":"2024-09-06T14:24:55.786032+0000","last_became_active":"2024-09-06T14:24:25.997614+0000","last_became_peered":"2024-09-06T14:24:25.997614+0000","last_unstale":"2024-09-06T14:24:55.786032+0000","last_undegraded":"2024-09-06T14:24:55.786032+0000","last_fullsized":"2024-09-06T14:24:55.786032+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_clean_scrub_stamp":"2024-09-06T14:24:24.970251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-08T01:10:51.683384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"60'5","reported_seq":36,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.922035+0000","last_change":"2024-09-06T14:24:28.012417+0000","last_active":"2024-09-06T14:24:55.922035+0000","last_peered":"2024-09-06T14:24:55.922035+0000","last_clean":"2024-09-06T14:24:55.922035+0000","last_became_active":"2024-09-06T14:24:28.012080+0000","last_became_peered":"2024-09-06T14:24:28.012080+0000","last_unstale":"2024-09-06T14:24:55.922035+0000","last_undegraded":"2024-09-06T14:24:55.922035+0000","last_fullsized":"2024-09-06T14:24:55.922035+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_clean_scrub_stamp":"2024-09-06T14:24:26.984416+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T17:19:54.900883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":22,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.447179+0000","last_change":"2024-09-06T14:24:30.020516+0000","last_active":"2024-09-06T14:24:55.447179+0000","last_peered":"2024-09-06T14:24:55.447179+0000","last_clean":"2024-09-06T14:24:55.447179+0000","last_became_active":"2024-09-06T14:24:30.020075+0000","last_became_peered":"2024-09-06T14:24:30.020075+0000","last_unstale":"2024-09-06T14:24:55.447179+0000","last_undegraded":"2024-09-06T14:24:55.447179+0000","last_fullsized":"2024-09-06T14:24:55.447179+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_clean_scrub_stamp":"2024-09-06T14:24:28.992633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T21:35:05.091193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":18,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.785716+0000","last_change":"2024-09-06T14:24:32.032981+0000","last_active":"2024-09-06T14:24:55.785716+0000","last_peered":"2024-09-06T14:24:55.785716+0000","last_clean":"2024-09-06T14:24:55.785716+0000","last_became_active":"2024-09-06T14:24:32.032715+0000","last_became_peered":"2024-09-06T14:24:32.032715+0000","last_unstale":"2024-09-06T14:24:55.785716+0000","last_undegraded":"2024-09-06T14:24:55.785716+0000","last_fullsized":"2024-09-06T14:24:55.785716+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_clean_scrub_stamp":"2024-09-06T14:24:31.009248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T21:19:16.074402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":18,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.922184+0000","last_change":"2024-09-06T14:24:32.032799+0000","last_active":"2024-09-06T14:24:55.922184+0000","last_peered":"2024-09-06T14:24:55.922184+0000","last_clean":"2024-09-06T14:24:55.922184+0000","last_became_active":"2024-09-06T14:24:32.032526+0000","last_became_peered":"2024-09-06T14:24:32.032526+0000","last_unstale":"2024-09-06T14:24:55.922184+0000","last_undegraded":"2024-09-06T14:24:55.922184+0000","last_fullsized":"2024-09-06T14:24:55.922184+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_clean_scrub_stamp":"2024-09-06T14:24:31.009248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T23:59:07.742457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":30,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.450651+0000","last_change":"2024-09-06T14:24:26.002157+0000","last_active":"2024-09-06T14:24:55.450651+0000","last_peered":"2024-09-06T14:24:55.450651+0000","last_clean":"2024-09-06T14:24:55.450651+0000","last_became_active":"2024-09-06T14:24:26.001828+0000","last_became_peered":"2024-09-06T14:24:26.001828+0000","last_unstale":"2024-09-06T14:24:55.450651+0000","last_undegraded":"2024-09-06T14:24:55.450651+0000","last_fullsized":"2024-09-06T14:24:55.450651+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_clean_scrub_stamp":"2024-09-06T14:24:24.970251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T21:29:27.668185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"60'15","reported_seq":51,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.842565+0000","last_change":"2024-09-06T14:24:28.013488+0000","last_active":"2024-09-06T14:24:55.842565+0000","last_peered":"2024-09-06T14:24:55.842565+0000","last_clean":"2024-09-06T14:24:55.842565+0000","last_became_active":"2024-09-06T14:24:28.013189+0000","last_became_peered":"2024-09-06T14:24:28.013189+0000","last_unstale":"2024-09-06T14:24:55.842565+0000","last_undegraded":"2024-09-06T14:24:55.842565+0000","last_fullsized":"2024-09-06T14:24:55.842565+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_clean_scrub_stamp":"2024-09-06T14:24:26.984416+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T15:39:41.635207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":22,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.923110+0000","last_change":"2024-09-06T14:24:30.021029+0000","last_active":"2024-09-06T14:24:55.923110+0000","last_peered":"2024-09-06T14:24:55.923110+0000","last_clean":"2024-09-06T14:24:55.923110+0000","last_became_active":"2024-09-06T14:24:30.020732+0000","last_became_peered":"2024-09-06T14:24:30.020732+0000","last_unstale":"2024-09-06T14:24:55.923110+0000","last_undegraded":"2024-09-06T14:24:55.923110+0000","last_fullsized":"2024-09-06T14:24:55.923110+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_clean_scrub_stamp":"2024-09-06T14:24:28.992633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T22:25:24.936291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":18,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.451607+0000","last_change":"2024-09-06T14:24:32.036230+0000","last_active":"2024-09-06T14:24:55.451607+0000","last_peered":"2024-09-06T14:24:55.451607+0000","last_clean":"2024-09-06T14:24:55.451607+0000","last_became_active":"2024-09-06T14:24:32.035789+0000","last_became_peered":"2024-09-06T14:24:32.035789+0000","last_unstale":"2024-09-06T14:24:55.451607+0000","last_undegraded":"2024-09-06T14:24:55.451607+0000","last_fullsized":"2024-09-06T14:24:55.451607+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_clean_scrub_stamp":"2024-09-06T14:24:31.009248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T21:32:34.018012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":30,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.922898+0000","last_change":"2024-09-06T14:24:25.995904+0000","last_active":"2024-09-06T14:24:55.922898+0000","last_peered":"2024-09-06T14:24:55.922898+0000","last_clean":"2024-09-06T14:24:55.922898+0000","last_became_active":"2024-09-06T14:24:25.995503+0000","last_became_peered":"2024-09-06T14:24:25.995503+0000","last_unstale":"2024-09-06T14:24:55.922898+0000","last_undegraded":"2024-09-06T14:24:55.922898+0000","last_fullsized":"2024-09-06T14:24:55.922898+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_clean_scrub_stamp":"2024-09-06T14:24:24.970251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T21:55:52.128974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"60'12","reported_seq":49,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.451049+0000","last_change":"2024-09-06T14:24:28.017290+0000","last_active":"2024-09-06T14:24:55.451049+0000","last_peered":"2024-09-06T14:24:55.451049+0000","last_clean":"2024-09-06T14:24:55.451049+0000","last_became_active":"2024-09-06T14:24:28.016965+0000","last_became_peered":"2024-09-06T14:24:28.016965+0000","last_unstale":"2024-09-06T14:24:55.451049+0000","last_undegraded":"2024-09-06T14:24:55.451049+0000","last_fullsized":"2024-09-06T14:24:55.451049+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_clean_scrub_stamp":"2024-09-06T14:24:26.984416+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T23:50:21.062604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":22,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.922540+0000","last_change":"2024-09-06T14:24:30.018987+0000","last_active":"2024-09-06T14:24:55.922540+0000","last_peered":"2024-09-06T14:24:55.922540+0000","last_clean":"2024-09-06T14:24:55.922540+0000","last_became_active":"2024-09-06T14:24:30.018717+0000","last_became_peered":"2024-09-06T14:24:30.018717+0000","last_unstale":"2024-09-06T14:24:55.922540+0000","last_undegraded":"2024-09-06T14:24:55.922540+0000","last_fullsized":"2024-09-06T14:24:55.922540+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_clean_scrub_stamp":"2024-09-06T14:24:28.992633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-08T01:29:49.391209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"60'1","reported_seq":19,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.447576+0000","last_change":"2024-09-06T14:24:32.037082+0000","last_active":"2024-09-06T14:24:55.447576+0000","last_peered":"2024-09-06T14:24:55.447576+0000","last_clean":"2024-09-06T14:24:55.447576+0000","last_became_active":"2024-09-06T14:24:32.036782+0000","last_became_peered":"2024-09-06T14:24:32.036782+0000","last_unstale":"2024-09-06T14:24:55.447576+0000","last_undegraded":"2024-09-06T14:24:55.447576+0000","last_fullsized":"2024-09-06T14:24:55.447576+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_clean_scrub_stamp":"2024-09-06T14:24:31.009248+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-08T01:17:20.757338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"53'1","reported_seq":31,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.922669+0000","last_change":"2024-09-06T14:24:26.001573+0000","last_active":"2024-09-06T14:24:55.922669+0000","last_peered":"2024-09-06T14:24:55.922669+0000","last_clean":"2024-09-06T14:24:55.922669+0000","last_became_active":"2024-09-06T14:24:25.998395+0000","last_became_peered":"2024-09-06T14:24:25.998395+0000","last_unstale":"2024-09-06T14:24:55.922669+0000","last_undegraded":"2024-09-06T14:24:55.922669+0000","last_fullsized":"2024-09-06T14:24:55.922669+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_clean_scrub_stamp":"2024-09-06T14:24:24.970251+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T17:36:56.816449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"60'10","reported_seq":41,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.450981+0000","last_change":"2024-09-06T14:24:28.062422+0000","last_active":"2024-09-06T14:24:55.450981+0000","last_peered":"2024-09-06T14:24:55.450981+0000","last_clean":"2024-09-06T14:24:55.450981+0000","last_became_active":"2024-09-06T14:24:28.062093+0000","last_became_peered":"2024-09-06T14:24:28.062093+0000","last_unstale":"2024-09-06T14:24:55.450981+0000","last_undegraded":"2024-09-06T14:24:55.450981+0000","last_fullsized":"2024-09-06T14:24:55.450981+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_clean_scrub_stamp":"2024-09-06T14:24:26.984416+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T16:51:37.986872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"60'8","reported_seq":43,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:25:06.056629+0000","last_change":"2024-09-06T14:24:30.016621+0000","last_active":"2024-09-06T14:25:06.056629+0000","last_peered":"2024-09-06T14:25:06.056629+0000","last_clean":"2024-09-06T14:25:06.056629+0000","last_became_active":"2024-09-06T14:24:30.015760+0000","last_became_peered":"2024-09-06T14:24:30.015760+0000","last_unstale":"2024-09-06T14:25:06.056629+0000","last_undegraded":"2024-09-06T14:25:06.056629+0000","last_fullsized":"2024-09-06T14:25:06.056629+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_clean_scrub_stamp":"2024-09-06T14:24:28.992633+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T20:30:40.346775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"60'15","reported_seq":51,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.923369+0000","last_change":"2024-09-06T14:24:28.016592+0000","last_active":"2024-09-06T14:24:55.923369+0000","last_peered":"2024-09-06T14:24:55.923369+0000","last_clean":"2024-09-06T14:24:55.923369+0000","last_became_active":"2024-09-06T14:24:28.016287+0000","last_became_peered":"2024-09-06T14:24:28.016287+0000","last_unstale":"2024-09-06T14:24:55.923369+0000","last_undegraded":"2024-09-06T14:24:55.923369+0000","last_fullsized":"2024-09-06T14:24:55.923369+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_clean_scrub_stamp":"2024-09-06T14:24:26.984416+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-08T01:17:07.274863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":30,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.450852+0000","last_change":"2024-09-06T14:24:26.005250+0000","last_active":"2024-09-06T14:24:55.450852+0000","last_peered":"2024-09-06T14:24:55.450852+0000","last_clean":"2024-09-06T14:24:55.450852+0000","last_became_active":"2024-09-06T14:24:26.004932+0000","last_became_peered":"2024-09-06T14:24:26.004932+0000","last_unstale":"2024-09-06T14:24:55.450852+0000","last_undegraded":"2024-09-06T14:24:55.450852+0000","last_fullsized":"2024-09-06T14:24:55.450852+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_clean_scrub_stamp":"2024-09-06T14:24:24.970251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T15:12:03.667862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"60'8","reported_seq":40,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:25:06.056683+0000","last_change":"2024-09-06T14:24:30.011504+0000","last_active":"2024-09-06T14:25:06.056683+0000","last_peered":"2024-09-06T14:25:06.056683+0000","last_clean":"2024-09-06T14:25:06.056683+0000","last_became_active":"2024-09-06T14:24:30.011217+0000","last_became_peered":"2024-09-06T14:24:30.011217+0000","last_unstale":"2024-09-06T14:25:06.056683+0000","last_undegraded":"2024-09-06T14:25:06.056683+0000","last_fullsized":"2024-09-06T14:25:06.056683+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_clean_scrub_stamp":"2024-09-06T14:24:28.992633+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T22:14:52.780516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":18,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.785977+0000","last_change":"2024-09-06T14:24:32.034536+0000","last_active":"2024-09-06T14:24:55.785977+0000","last_peered":"2024-09-06T14:24:55.785977+0000","last_clean":"2024-09-06T14:24:55.785977+0000","last_became_active":"2024-09-06T14:24:32.034128+0000","last_became_peered":"2024-09-06T14:24:32.034128+0000","last_unstale":"2024-09-06T14:24:55.785977+0000","last_undegraded":"2024-09-06T14:24:55.785977+0000","last_fullsized":"2024-09-06T14:24:55.785977+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_clean_scrub_stamp":"2024-09-06T14:24:31.009248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T20:10:18.987767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.0","version":"60'18","reported_seq":58,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.450596+0000","last_change":"2024-09-06T14:24:28.064719+0000","last_active":"2024-09-06T14:24:55.450596+0000","last_peered":"2024-09-06T14:24:55.450596+0000","last_clean":"2024-09-06T14:24:55.450596+0000","last_became_active":"2024-09-06T14:24:28.064397+0000","last_became_peered":"2024-09-06T14:24:28.064397+0000","last_unstale":"2024-09-06T14:24:55.450596+0000","last_undegraded":"2024-09-06T14:24:55.450596+0000","last_fullsized":"2024-09-06T14:24:55.450596+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_clean_scrub_stamp":"2024-09-06T14:24:26.984416+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T21:56:32.478236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":30,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.452061+0000","last_change":"2024-09-06T14:24:26.003024+0000","last_active":"2024-09-06T14:24:55.452061+0000","last_peered":"2024-09-06T14:24:55.452061+0000","last_clean":"2024-09-06T14:24:55.452061+0000","last_became_active":"2024-09-06T14:24:26.002756+0000","last_became_peered":"2024-09-06T14:24:26.002756+0000","last_unstale":"2024-09-06T14:24:55.452061+0000","last_undegraded":"2024-09-06T14:24:55.452061+0000","last_fullsized":"2024-09-06T14:24:55.452061+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_clean_scrub_stamp":"2024-09-06T14:24:24.970251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T22:43:13.116583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":22,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.921587+0000","last_change":"2024-09-06T14:24:30.014409+0000","last_active":"2024-09-06T14:24:55.921587+0000","last_peered":"2024-09-06T14:24:55.921587+0000","last_clean":"2024-09-06T14:24:55.921587+0000","last_became_active":"2024-09-06T14:24:30.014140+0000","last_became_peered":"2024-09-06T14:24:30.014140+0000","last_unstale":"2024-09-06T14:24:55.921587+0000","last_undegraded":"2024-09-06T14:24:55.921587+0000","last_fullsized":"2024-09-06T14:24:55.921587+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_clean_scrub_stamp":"2024-09-06T14:24:28.992633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T23:53:18.617224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":18,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.922680+0000","last_change":"2024-09-06T14:24:32.036763+0000","last_active":"2024-09-06T14:24:55.922680+0000","last_peered":"2024-09-06T14:24:55.922680+0000","last_clean":"2024-09-06T14:24:55.922680+0000","last_became_active":"2024-09-06T14:24:32.036482+0000","last_became_peered":"2024-09-06T14:24:32.036482+0000","last_unstale":"2024-09-06T14:24:55.922680+0000","last_undegraded":"2024-09-06T14:24:55.922680+0000","last_fullsized":"2024-09-06T14:24:55.922680+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_clean_scrub_stamp":"2024-09-06T14:24:31.009248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T20:56:49.936018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"60'14","reported_seq":47,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.921681+0000","last_change":"2024-09-06T14:24:28.002153+0000","last_active":"2024-09-06T14:24:55.921681+0000","last_peered":"2024-09-06T14:24:55.921681+0000","last_clean":"2024-09-06T14:24:55.921681+0000","last_became_active":"2024-09-06T14:24:28.001864+0000","last_became_peered":"2024-09-06T14:24:28.001864+0000","last_unstale":"2024-09-06T14:24:55.921681+0000","last_undegraded":"2024-09-06T14:24:55.921681+0000","last_fullsized":"2024-09-06T14:24:55.921681+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_clean_scrub_stamp":"2024-09-06T14:24:26.984416+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T17:02:53.838105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"53'1","reported_seq":31,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.451342+0000","last_change":"2024-09-06T14:24:25.999108+0000","last_active":"2024-09-06T14:24:55.451342+0000","last_peered":"2024-09-06T14:24:55.451342+0000","last_clean":"2024-09-06T14:24:55.451342+0000","last_became_active":"2024-09-06T14:24:25.998791+0000","last_became_peered":"2024-09-06T14:24:25.998791+0000","last_unstale":"2024-09-06T14:24:55.451342+0000","last_undegraded":"2024-09-06T14:24:55.451342+0000","last_fullsized":"2024-09-06T14:24:55.451342+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_clean_scrub_stamp":"2024-09-06T14:24:24.970251+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T22:56:31.483325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"60'8","reported_seq":40,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:25:06.056425+0000","last_change":"2024-09-06T14:24:30.014569+0000","last_active":"2024-09-06T14:25:06.056425+0000","last_peered":"2024-09-06T14:25:06.056425+0000","last_clean":"2024-09-06T14:25:06.056425+0000","last_became_active":"2024-09-06T14:24:30.014272+0000","last_became_peered":"2024-09-06T14:24:30.014272+0000","last_unstale":"2024-09-06T14:25:06.056425+0000","last_undegraded":"2024-09-06T14:25:06.056425+0000","last_fullsized":"2024-09-06T14:25:06.056425+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_clean_scrub_stamp":"2024-09-06T14:24:28.992633+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-08T01:08:12.708501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":18,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.447320+0000","last_change":"2024-09-06T14:24:32.032068+0000","last_active":"2024-09-06T14:24:55.447320+0000","last_peered":"2024-09-06T14:24:55.447320+0000","last_clean":"2024-09-06T14:24:55.447320+0000","last_became_active":"2024-09-06T14:24:32.031785+0000","last_became_peered":"2024-09-06T14:24:32.031785+0000","last_unstale":"2024-09-06T14:24:55.447320+0000","last_undegraded":"2024-09-06T14:24:55.447320+0000","last_fullsized":"2024-09-06T14:24:55.447320+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_clean_scrub_stamp":"2024-09-06T14:24:31.009248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T18:47:27.357801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"60'10","reported_seq":41,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.922780+0000","last_change":"2024-09-06T14:24:28.005612+0000","last_active":"2024-09-06T14:24:55.922780+0000","last_peered":"2024-09-06T14:24:55.922780+0000","last_clean":"2024-09-06T14:24:55.922780+0000","last_became_active":"2024-09-06T14:24:28.005344+0000","last_became_peered":"2024-09-06T14:24:28.005344+0000","last_unstale":"2024-09-06T14:24:55.922780+0000","last_undegraded":"2024-09-06T14:24:55.922780+0000","last_fullsized":"2024-09-06T14:24:55.922780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_clean_scrub_stamp":"2024-09-06T14:24:26.984416+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T20:08:41.402288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":30,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.786295+0000","last_change":"2024-09-06T14:24:26.000038+0000","last_active":"2024-09-06T14:24:55.786295+0000","last_peered":"2024-09-06T14:24:55.786295+0000","last_clean":"2024-09-06T14:24:55.786295+0000","last_became_active":"2024-09-06T14:24:25.999753+0000","last_became_peered":"2024-09-06T14:24:25.999753+0000","last_unstale":"2024-09-06T14:24:55.786295+0000","last_undegraded":"2024-09-06T14:24:55.786295+0000","last_fullsized":"2024-09-06T14:24:55.786295+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_clean_scrub_stamp":"2024-09-06T14:24:24.970251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T16:18:07.617375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"60'8","reported_seq":43,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:25:06.056381+0000","last_change":"2024-09-06T14:24:30.014339+0000","last_active":"2024-09-06T14:25:06.056381+0000","last_peered":"2024-09-06T14:25:06.056381+0000","last_clean":"2024-09-06T14:25:06.056381+0000","last_became_active":"2024-09-06T14:24:30.014022+0000","last_became_peered":"2024-09-06T14:24:30.014022+0000","last_unstale":"2024-09-06T14:25:06.056381+0000","last_undegraded":"2024-09-06T14:25:06.056381+0000","last_fullsized":"2024-09-06T14:25:06.056381+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_clean_scrub_stamp":"2024-09-06T14:24:28.992633+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T20:24:25.037122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":18,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.450449+0000","last_change":"2024-09-06T14:24:32.033170+0000","last_active":"2024-09-06T14:24:55.450449+0000","last_peered":"2024-09-06T14:24:55.450449+0000","last_clean":"2024-09-06T14:24:55.450449+0000","last_became_active":"2024-09-06T14:24:32.032844+0000","last_became_peered":"2024-09-06T14:24:32.032844+0000","last_unstale":"2024-09-06T14:24:55.450449+0000","last_undegraded":"2024-09-06T14:24:55.450449+0000","last_fullsized":"2024-09-06T14:24:55.450449+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_clean_scrub_stamp":"2024-09-06T14:24:31.009248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T18:07:13.387892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"60'19","reported_seq":62,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.450359+0000","last_change":"2024-09-06T14:24:28.062441+0000","last_active":"2024-09-06T14:24:55.450359+0000","last_peered":"2024-09-06T14:24:55.450359+0000","last_clean":"2024-09-06T14:24:55.450359+0000","last_became_active":"2024-09-06T14:24:28.062131+0000","last_became_peered":"2024-09-06T14:24:28.062131+0000","last_unstale":"2024-09-06T14:24:55.450359+0000","last_undegraded":"2024-09-06T14:24:55.450359+0000","last_fullsized":"2024-09-06T14:24:55.450359+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_clean_scrub_stamp":"2024-09-06T14:24:26.984416+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T20:58:22.084533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":30,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.923231+0000","last_change":"2024-09-06T14:24:25.993201+0000","last_active":"2024-09-06T14:24:55.923231+0000","last_peered":"2024-09-06T14:24:55.923231+0000","last_clean":"2024-09-06T14:24:55.923231+0000","last_became_active":"2024-09-06T14:24:25.992980+0000","last_became_peered":"2024-09-06T14:24:25.992980+0000","last_unstale":"2024-09-06T14:24:55.923231+0000","last_undegraded":"2024-09-06T14:24:55.923231+0000","last_fullsized":"2024-09-06T14:24:55.923231+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_clean_scrub_stamp":"2024-09-06T14:24:24.970251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T19:24:50.416704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":22,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.921859+0000","last_change":"2024-09-06T14:24:30.024052+0000","last_active":"2024-09-06T14:24:55.921859+0000","last_peered":"2024-09-06T14:24:55.921859+0000","last_clean":"2024-09-06T14:24:55.921859+0000","last_became_active":"2024-09-06T14:24:30.023745+0000","last_became_peered":"2024-09-06T14:24:30.023745+0000","last_unstale":"2024-09-06T14:24:55.921859+0000","last_undegraded":"2024-09-06T14:24:55.921859+0000","last_fullsized":"2024-09-06T14:24:55.921859+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_clean_scrub_stamp":"2024-09-06T14:24:28.992633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T17:42:37.103554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":18,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.922312+0000","last_change":"2024-09-06T14:24:32.031320+0000","last_active":"2024-09-06T14:24:55.922312+0000","last_peered":"2024-09-06T14:24:55.922312+0000","last_clean":"2024-09-06T14:24:55.922312+0000","last_became_active":"2024-09-06T14:24:32.030500+0000","last_became_peered":"2024-09-06T14:24:32.030500+0000","last_unstale":"2024-09-06T14:24:55.922312+0000","last_undegraded":"2024-09-06T14:24:55.922312+0000","last_fullsized":"2024-09-06T14:24:55.922312+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_clean_scrub_stamp":"2024-09-06T14:24:31.009248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T20:28:34.604094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"60'28","reported_seq":84,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:25:06.056434+0000","last_change":"2024-09-06T14:24:28.015353+0000","last_active":"2024-09-06T14:25:06.056434+0000","last_peered":"2024-09-06T14:25:06.056434+0000","last_clean":"2024-09-06T14:25:06.056434+0000","last_became_active":"2024-09-06T14:24:28.015055+0000","last_became_peered":"2024-09-06T14:24:28.015055+0000","last_unstale":"2024-09-06T14:25:06.056434+0000","last_undegraded":"2024-09-06T14:25:06.056434+0000","last_fullsized":"2024-09-06T14:25:06.056434+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_clean_scrub_stamp":"2024-09-06T14:24:26.984416+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T21:28:19.010695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":30,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.922733+0000","last_change":"2024-09-06T14:24:26.001741+0000","last_active":"2024-09-06T14:24:55.922733+0000","last_peered":"2024-09-06T14:24:55.922733+0000","last_clean":"2024-09-06T14:24:55.922733+0000","last_became_active":"2024-09-06T14:24:26.001177+0000","last_became_peered":"2024-09-06T14:24:26.001177+0000","last_unstale":"2024-09-06T14:24:55.922733+0000","last_undegraded":"2024-09-06T14:24:55.922733+0000","last_fullsized":"2024-09-06T14:24:55.922733+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_clean_scrub_stamp":"2024-09-06T14:24:24.970251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T16:10:12.245622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"55'2","reported_seq":40,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.785920+0000","last_change":"2024-09-06T14:24:28.001551+0000","last_active":"2024-09-06T14:24:55.785920+0000","last_peered":"2024-09-06T14:24:55.785920+0000","last_clean":"2024-09-06T14:24:55.785920+0000","last_became_active":"2024-09-06T14:24:24.990939+0000","last_became_peered":"2024-09-06T14:24:24.990939+0000","last_unstale":"2024-09-06T14:24:55.785920+0000","last_undegraded":"2024-09-06T14:24:55.785920+0000","last_fullsized":"2024-09-06T14:24:55.785920+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:23.962291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:23.962291+0000","last_clean_scrub_stamp":"2024-09-06T14:24:23.962291+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T17:12:52.384373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011799650000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":22,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.449453+0000","last_change":"2024-09-06T14:24:30.015733+0000","last_active":"2024-09-06T14:24:55.449453+0000","last_peered":"2024-09-06T14:24:55.449453+0000","last_clean":"2024-09-06T14:24:55.449453+0000","last_became_active":"2024-09-06T14:24:30.015448+0000","last_became_peered":"2024-09-06T14:24:30.015448+0000","last_unstale":"2024-09-06T14:24:55.449453+0000","last_undegraded":"2024-09-06T14:24:55.449453+0000","last_fullsized":"2024-09-06T14:24:55.449453+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_clean_scrub_stamp":"2024-09-06T14:24:28.992633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T14:57:20.026461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":18,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.451325+0000","last_change":"2024-09-06T14:24:32.038106+0000","last_active":"2024-09-06T14:24:55.451325+0000","last_peered":"2024-09-06T14:24:55.451325+0000","last_clean":"2024-09-06T14:24:55.451325+0000","last_became_active":"2024-09-06T14:24:32.037762+0000","last_became_peered":"2024-09-06T14:24:32.037762+0000","last_unstale":"2024-09-06T14:24:55.451325+0000","last_undegraded":"2024-09-06T14:24:55.451325+0000","last_fullsized":"2024-09-06T14:24:55.451325+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_clean_scrub_stamp":"2024-09-06T14:24:31.009248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T22:25:36.424591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"60'13","reported_seq":53,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.923429+0000","last_change":"2024-09-06T14:24:28.007350+0000","last_active":"2024-09-06T14:24:55.923429+0000","last_peered":"2024-09-06T14:24:55.923429+0000","last_clean":"2024-09-06T14:24:55.923429+0000","last_became_active":"2024-09-06T14:24:28.007105+0000","last_became_peered":"2024-09-06T14:24:28.007105+0000","last_unstale":"2024-09-06T14:24:55.923429+0000","last_undegraded":"2024-09-06T14:24:55.923429+0000","last_fullsized":"2024-09-06T14:24:55.923429+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_clean_scrub_stamp":"2024-09-06T14:24:26.984416+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T19:11:11.276341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":30,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.922311+0000","last_change":"2024-09-06T14:24:25.988258+0000","last_active":"2024-09-06T14:24:55.922311+0000","last_peered":"2024-09-06T14:24:55.922311+0000","last_clean":"2024-09-06T14:24:55.922311+0000","last_became_active":"2024-09-06T14:24:25.987935+0000","last_became_peered":"2024-09-06T14:24:25.987935+0000","last_unstale":"2024-09-06T14:24:55.922311+0000","last_undegraded":"2024-09-06T14:24:55.922311+0000","last_fullsized":"2024-09-06T14:24:55.922311+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_clean_scrub_stamp":"2024-09-06T14:24:24.970251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T18:18:47.349222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"53'1","reported_seq":39,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.841772+0000","last_change":"2024-09-06T14:24:28.005058+0000","last_active":"2024-09-06T14:24:55.841772+0000","last_peered":"2024-09-06T14:24:55.841772+0000","last_clean":"2024-09-06T14:24:55.841772+0000","last_became_active":"2024-09-06T14:24:24.987965+0000","last_became_peered":"2024-09-06T14:24:24.987965+0000","last_unstale":"2024-09-06T14:24:55.841772+0000","last_undegraded":"2024-09-06T14:24:55.841772+0000","last_fullsized":"2024-09-06T14:24:55.841772+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:23.962291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:23.962291+0000","last_clean_scrub_stamp":"2024-09-06T14:24:23.962291+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T23:31:31.658900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0046516300000000003,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":22,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.842325+0000","last_change":"2024-09-06T14:24:30.019424+0000","last_active":"2024-09-06T14:24:55.842325+0000","last_peered":"2024-09-06T14:24:55.842325+0000","last_clean":"2024-09-06T14:24:55.842325+0000","last_became_active":"2024-09-06T14:24:30.019111+0000","last_became_peered":"2024-09-06T14:24:30.019111+0000","last_unstale":"2024-09-06T14:24:55.842325+0000","last_undegraded":"2024-09-06T14:24:55.842325+0000","last_fullsized":"2024-09-06T14:24:55.842325+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_clean_scrub_stamp":"2024-09-06T14:24:28.992633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T20:06:38.974950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":18,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.447811+0000","last_change":"2024-09-06T14:24:32.040295+0000","last_active":"2024-09-06T14:24:55.447811+0000","last_peered":"2024-09-06T14:24:55.447811+0000","last_clean":"2024-09-06T14:24:55.447811+0000","last_became_active":"2024-09-06T14:24:32.039953+0000","last_became_peered":"2024-09-06T14:24:32.039953+0000","last_unstale":"2024-09-06T14:24:55.447811+0000","last_undegraded":"2024-09-06T14:24:55.447811+0000","last_fullsized":"2024-09-06T14:24:55.447811+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_clean_scrub_stamp":"2024-09-06T14:24:31.009248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-08T00:45:52.313889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"60'12","reported_seq":44,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.451102+0000","last_change":"2024-09-06T14:24:28.008746+0000","last_active":"2024-09-06T14:24:55.451102+0000","last_peered":"2024-09-06T14:24:55.451102+0000","last_clean":"2024-09-06T14:24:55.451102+0000","last_became_active":"2024-09-06T14:24:28.008441+0000","last_became_peered":"2024-09-06T14:24:28.008441+0000","last_unstale":"2024-09-06T14:24:55.451102+0000","last_undegraded":"2024-09-06T14:24:55.451102+0000","last_fullsized":"2024-09-06T14:24:55.451102+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_clean_scrub_stamp":"2024-09-06T14:24:26.984416+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-08T02:14:08.659425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":30,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.448885+0000","last_change":"2024-09-06T14:24:26.005124+0000","last_active":"2024-09-06T14:24:55.448885+0000","last_peered":"2024-09-06T14:24:55.448885+0000","last_clean":"2024-09-06T14:24:55.448885+0000","last_became_active":"2024-09-06T14:24:26.004836+0000","last_became_peered":"2024-09-06T14:24:26.004836+0000","last_unstale":"2024-09-06T14:24:55.448885+0000","last_undegraded":"2024-09-06T14:24:55.448885+0000","last_fullsized":"2024-09-06T14:24:55.448885+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_clean_scrub_stamp":"2024-09-06T14:24:24.970251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T18:23:00.702043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"61'5","reported_seq":81,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:25:10.033064+0000","last_change":"2024-09-06T14:24:28.062498+0000","last_active":"2024-09-06T14:25:10.033064+0000","last_peered":"2024-09-06T14:25:10.033064+0000","last_clean":"2024-09-06T14:25:10.033064+0000","last_became_active":"2024-09-06T14:24:24.991638+0000","last_became_peered":"2024-09-06T14:24:24.991638+0000","last_unstale":"2024-09-06T14:25:10.033064+0000","last_undegraded":"2024-09-06T14:25:10.033064+0000","last_fullsized":"2024-09-06T14:25:10.033064+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:23.962291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:23.962291+0000","last_clean_scrub_stamp":"2024-09-06T14:24:23.962291+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T22:07:02.083358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061857000000000004,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":40,"num_read_kb":35,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":22,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.786384+0000","last_change":"2024-09-06T14:24:30.019269+0000","last_active":"2024-09-06T14:24:55.786384+0000","last_peered":"2024-09-06T14:24:55.786384+0000","last_clean":"2024-09-06T14:24:55.786384+0000","last_became_active":"2024-09-06T14:24:30.018880+0000","last_became_peered":"2024-09-06T14:24:30.018880+0000","last_unstale":"2024-09-06T14:24:55.786384+0000","last_undegraded":"2024-09-06T14:24:55.786384+0000","last_fullsized":"2024-09-06T14:24:55.786384+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_clean_scrub_stamp":"2024-09-06T14:24:28.992633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T19:29:09.374819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":18,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.922818+0000","last_change":"2024-09-06T14:24:32.034342+0000","last_active":"2024-09-06T14:24:55.922818+0000","last_peered":"2024-09-06T14:24:55.922818+0000","last_clean":"2024-09-06T14:24:55.922818+0000","last_became_active":"2024-09-06T14:24:32.034023+0000","last_became_peered":"2024-09-06T14:24:32.034023+0000","last_unstale":"2024-09-06T14:24:55.922818+0000","last_undegraded":"2024-09-06T14:24:55.922818+0000","last_fullsized":"2024-09-06T14:24:55.922818+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_clean_scrub_stamp":"2024-09-06T14:24:31.009248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T18:03:07.564884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"60'16","reported_seq":58,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:25:06.056689+0000","last_change":"2024-09-06T14:24:28.062894+0000","last_active":"2024-09-06T14:25:06.056689+0000","last_peered":"2024-09-06T14:25:06.056689+0000","last_clean":"2024-09-06T14:25:06.056689+0000","last_became_active":"2024-09-06T14:24:28.062494+0000","last_became_peered":"2024-09-06T14:24:28.062494+0000","last_unstale":"2024-09-06T14:25:06.056689+0000","last_undegraded":"2024-09-06T14:25:06.056689+0000","last_fullsized":"2024-09-06T14:25:06.056689+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_clean_scrub_stamp":"2024-09-06T14:24:26.984416+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T18:15:25.399135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":30,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.450048+0000","last_change":"2024-09-06T14:24:26.000957+0000","last_active":"2024-09-06T14:24:55.450048+0000","last_peered":"2024-09-06T14:24:55.450048+0000","last_clean":"2024-09-06T14:24:55.450048+0000","last_became_active":"2024-09-06T14:24:26.000658+0000","last_became_peered":"2024-09-06T14:24:26.000658+0000","last_unstale":"2024-09-06T14:24:55.450048+0000","last_undegraded":"2024-09-06T14:24:55.450048+0000","last_fullsized":"2024-09-06T14:24:55.450048+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_clean_scrub_stamp":"2024-09-06T14:24:24.970251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T18:22:03.206317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"62'79","reported_seq":67,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:57.815056+0000","last_change":"2024-09-06T14:24:19.176211+0000","last_active":"2024-09-06T14:24:57.815056+0000","last_peered":"2024-09-06T14:24:57.815056+0000","last_clean":"2024-09-06T14:24:57.815056+0000","last_became_active":"2024-09-06T14:24:19.169398+0000","last_became_peered":"2024-09-06T14:24:19.169398+0000","last_unstale":"2024-09-06T14:24:57.815056+0000","last_undegraded":"2024-09-06T14:24:57.815056+0000","last_fullsized":"2024-09-06T14:24:57.815056+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:22:11.956383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:22:11.956383+0000","last_clean_scrub_stamp":"2024-09-06T14:22:11.956383+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T16:04:11.762037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":22,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.447706+0000","last_change":"2024-09-06T14:24:30.021108+0000","last_active":"2024-09-06T14:24:55.447706+0000","last_peered":"2024-09-06T14:24:55.447706+0000","last_clean":"2024-09-06T14:24:55.447706+0000","last_became_active":"2024-09-06T14:24:30.020796+0000","last_became_peered":"2024-09-06T14:24:30.020796+0000","last_unstale":"2024-09-06T14:24:55.447706+0000","last_undegraded":"2024-09-06T14:24:55.447706+0000","last_fullsized":"2024-09-06T14:24:55.447706+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_clean_scrub_stamp":"2024-09-06T14:24:28.992633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T23:21:45.657726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":18,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.786317+0000","last_change":"2024-09-06T14:24:32.035497+0000","last_active":"2024-09-06T14:24:55.786317+0000","last_peered":"2024-09-06T14:24:55.786317+0000","last_clean":"2024-09-06T14:24:55.786317+0000","last_became_active":"2024-09-06T14:24:32.034752+0000","last_became_peered":"2024-09-06T14:24:32.034752+0000","last_unstale":"2024-09-06T14:24:55.786317+0000","last_undegraded":"2024-09-06T14:24:55.786317+0000","last_fullsized":"2024-09-06T14:24:55.786317+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_clean_scrub_stamp":"2024-09-06T14:24:31.009248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T23:19:54.020180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"60'11","reported_seq":45,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.922959+0000","last_change":"2024-09-06T14:24:28.004795+0000","last_active":"2024-09-06T14:24:55.922959+0000","last_peered":"2024-09-06T14:24:55.922959+0000","last_clean":"2024-09-06T14:24:55.922959+0000","last_became_active":"2024-09-06T14:24:28.004411+0000","last_became_peered":"2024-09-06T14:24:28.004411+0000","last_unstale":"2024-09-06T14:24:55.922959+0000","last_undegraded":"2024-09-06T14:24:55.922959+0000","last_fullsized":"2024-09-06T14:24:55.922959+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_clean_scrub_stamp":"2024-09-06T14:24:26.984416+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T17:15:55.813063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":30,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.922131+0000","last_change":"2024-09-06T14:24:26.000813+0000","last_active":"2024-09-06T14:24:55.922131+0000","last_peered":"2024-09-06T14:24:55.922131+0000","last_clean":"2024-09-06T14:24:55.922131+0000","last_became_active":"2024-09-06T14:24:26.000247+0000","last_became_peered":"2024-09-06T14:24:26.000247+0000","last_unstale":"2024-09-06T14:24:55.922131+0000","last_undegraded":"2024-09-06T14:24:55.922131+0000","last_fullsized":"2024-09-06T14:24:55.922131+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_clean_scrub_stamp":"2024-09-06T14:24:24.970251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T19:34:42.146507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":22,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.786223+0000","last_change":"2024-09-06T14:24:30.013658+0000","last_active":"2024-09-06T14:24:55.786223+0000","last_peered":"2024-09-06T14:24:55.786223+0000","last_clean":"2024-09-06T14:24:55.786223+0000","last_became_active":"2024-09-06T14:24:30.013334+0000","last_became_peered":"2024-09-06T14:24:30.013334+0000","last_unstale":"2024-09-06T14:24:55.786223+0000","last_undegraded":"2024-09-06T14:24:55.786223+0000","last_fullsized":"2024-09-06T14:24:55.786223+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_clean_scrub_stamp":"2024-09-06T14:24:28.992633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T21:28:56.142218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":18,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.450135+0000","last_change":"2024-09-06T14:24:32.035785+0000","last_active":"2024-09-06T14:24:55.450135+0000","last_peered":"2024-09-06T14:24:55.450135+0000","last_clean":"2024-09-06T14:24:55.450135+0000","last_became_active":"2024-09-06T14:24:32.035484+0000","last_became_peered":"2024-09-06T14:24:32.035484+0000","last_unstale":"2024-09-06T14:24:55.450135+0000","last_undegraded":"2024-09-06T14:24:55.450135+0000","last_fullsized":"2024-09-06T14:24:55.450135+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_clean_scrub_stamp":"2024-09-06T14:24:31.009248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-08T00:29:42.639305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"60'17","reported_seq":54,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.922370+0000","last_change":"2024-09-06T14:24:28.008597+0000","last_active":"2024-09-06T14:24:55.922370+0000","last_peered":"2024-09-06T14:24:55.922370+0000","last_clean":"2024-09-06T14:24:55.922370+0000","last_became_active":"2024-09-06T14:24:28.008244+0000","last_became_peered":"2024-09-06T14:24:28.008244+0000","last_unstale":"2024-09-06T14:24:55.922370+0000","last_undegraded":"2024-09-06T14:24:55.922370+0000","last_fullsized":"2024-09-06T14:24:55.922370+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_clean_scrub_stamp":"2024-09-06T14:24:26.984416+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-08T00:38:37.360978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"53'1","reported_seq":36,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.922301+0000","last_change":"2024-09-06T14:24:25.994258+0000","last_active":"2024-09-06T14:24:55.922301+0000","last_peered":"2024-09-06T14:24:55.922301+0000","last_clean":"2024-09-06T14:24:55.922301+0000","last_became_active":"2024-09-06T14:24:25.993164+0000","last_became_peered":"2024-09-06T14:24:25.993164+0000","last_unstale":"2024-09-06T14:24:55.922301+0000","last_undegraded":"2024-09-06T14:24:55.922301+0000","last_fullsized":"2024-09-06T14:24:55.922301+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_clean_scrub_stamp":"2024-09-06T14:24:24.970251+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T23:16:28.120890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":22,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.922909+0000","last_change":"2024-09-06T14:24:30.020695+0000","last_active":"2024-09-06T14:24:55.922909+0000","last_peered":"2024-09-06T14:24:55.922909+0000","last_clean":"2024-09-06T14:24:55.922909+0000","last_became_active":"2024-09-06T14:24:30.020195+0000","last_became_peered":"2024-09-06T14:24:30.020195+0000","last_unstale":"2024-09-06T14:24:55.922909+0000","last_undegraded":"2024-09-06T14:24:55.922909+0000","last_fullsized":"2024-09-06T14:24:55.922909+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_clean_scrub_stamp":"2024-09-06T14:24:28.992633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-08T01:13:58.867798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":18,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.842219+0000","last_change":"2024-09-06T14:24:32.034092+0000","last_active":"2024-09-06T14:24:55.842219+0000","last_peered":"2024-09-06T14:24:55.842219+0000","last_clean":"2024-09-06T14:24:55.842219+0000","last_became_active":"2024-09-06T14:24:32.033678+0000","last_became_peered":"2024-09-06T14:24:32.033678+0000","last_unstale":"2024-09-06T14:24:55.842219+0000","last_undegraded":"2024-09-06T14:24:55.842219+0000","last_fullsized":"2024-09-06T14:24:55.842219+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_clean_scrub_stamp":"2024-09-06T14:24:31.009248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T23:11:45.663142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"60'10","reported_seq":41,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.922634+0000","last_change":"2024-09-06T14:24:28.013847+0000","last_active":"2024-09-06T14:24:55.922634+0000","last_peered":"2024-09-06T14:24:55.922634+0000","last_clean":"2024-09-06T14:24:55.922634+0000","last_became_active":"2024-09-06T14:24:28.013287+0000","last_became_peered":"2024-09-06T14:24:28.013287+0000","last_unstale":"2024-09-06T14:24:55.922634+0000","last_undegraded":"2024-09-06T14:24:55.922634+0000","last_fullsized":"2024-09-06T14:24:55.922634+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_clean_scrub_stamp":"2024-09-06T14:24:26.984416+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T15:18:46.754706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":30,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.450173+0000","last_change":"2024-09-06T14:24:26.003545+0000","last_active":"2024-09-06T14:24:55.450173+0000","last_peered":"2024-09-06T14:24:55.450173+0000","last_clean":"2024-09-06T14:24:55.450173+0000","last_became_active":"2024-09-06T14:24:26.003278+0000","last_became_peered":"2024-09-06T14:24:26.003278+0000","last_unstale":"2024-09-06T14:24:55.450173+0000","last_undegraded":"2024-09-06T14:24:55.450173+0000","last_fullsized":"2024-09-06T14:24:55.450173+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_clean_scrub_stamp":"2024-09-06T14:24:24.970251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-08T01:01:43.104524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"60'8","reported_seq":40,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:25:06.056414+0000","last_change":"2024-09-06T14:24:30.021484+0000","last_active":"2024-09-06T14:25:06.056414+0000","last_peered":"2024-09-06T14:25:06.056414+0000","last_clean":"2024-09-06T14:25:06.056414+0000","last_became_active":"2024-09-06T14:24:30.021129+0000","last_became_peered":"2024-09-06T14:24:30.021129+0000","last_unstale":"2024-09-06T14:25:06.056414+0000","last_undegraded":"2024-09-06T14:25:06.056414+0000","last_fullsized":"2024-09-06T14:25:06.056414+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_clean_scrub_stamp":"2024-09-06T14:24:28.992633+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T23:03:00.443813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"60'1","reported_seq":19,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.922058+0000","last_change":"2024-09-06T14:24:32.030737+0000","last_active":"2024-09-06T14:24:55.922058+0000","last_peered":"2024-09-06T14:24:55.922058+0000","last_clean":"2024-09-06T14:24:55.922058+0000","last_became_active":"2024-09-06T14:24:32.030434+0000","last_became_peered":"2024-09-06T14:24:32.030434+0000","last_unstale":"2024-09-06T14:24:55.922058+0000","last_undegraded":"2024-09-06T14:24:55.922058+0000","last_fullsized":"2024-09-06T14:24:55.922058+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_clean_scrub_stamp":"2024-09-06T14:24:31.009248+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T16:21:06.363040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"60'9","reported_seq":42,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.450234+0000","last_change":"2024-09-06T14:24:28.005629+0000","last_active":"2024-09-06T14:24:55.450234+0000","last_peered":"2024-09-06T14:24:55.450234+0000","last_clean":"2024-09-06T14:24:55.450234+0000","last_became_active":"2024-09-06T14:24:28.005238+0000","last_became_peered":"2024-09-06T14:24:28.005238+0000","last_unstale":"2024-09-06T14:24:55.450234+0000","last_undegraded":"2024-09-06T14:24:55.450234+0000","last_fullsized":"2024-09-06T14:24:55.450234+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_clean_scrub_stamp":"2024-09-06T14:24:26.984416+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T21:10:02.684938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":30,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.786105+0000","last_change":"2024-09-06T14:24:25.999292+0000","last_active":"2024-09-06T14:24:55.786105+0000","last_peered":"2024-09-06T14:24:55.786105+0000","last_clean":"2024-09-06T14:24:55.786105+0000","last_became_active":"2024-09-06T14:24:25.999018+0000","last_became_peered":"2024-09-06T14:24:25.999018+0000","last_unstale":"2024-09-06T14:24:55.786105+0000","last_undegraded":"2024-09-06T14:24:55.786105+0000","last_fullsized":"2024-09-06T14:24:55.786105+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_clean_scrub_stamp":"2024-09-06T14:24:24.970251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T20:23:44.349519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":22,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.841713+0000","last_change":"2024-09-06T14:24:30.020394+0000","last_active":"2024-09-06T14:24:55.841713+0000","last_peered":"2024-09-06T14:24:55.841713+0000","last_clean":"2024-09-06T14:24:55.841713+0000","last_became_active":"2024-09-06T14:24:30.020089+0000","last_became_peered":"2024-09-06T14:24:30.020089+0000","last_unstale":"2024-09-06T14:24:55.841713+0000","last_undegraded":"2024-09-06T14:24:55.841713+0000","last_fullsized":"2024-09-06T14:24:55.841713+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_clean_scrub_stamp":"2024-09-06T14:24:28.992633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-08T00:30:08.037871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":18,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.448682+0000","last_change":"2024-09-06T14:24:32.032306+0000","last_active":"2024-09-06T14:24:55.448682+0000","last_peered":"2024-09-06T14:24:55.448682+0000","last_clean":"2024-09-06T14:24:55.448682+0000","last_became_active":"2024-09-06T14:24:32.032004+0000","last_became_peered":"2024-09-06T14:24:32.032004+0000","last_unstale":"2024-09-06T14:24:55.448682+0000","last_undegraded":"2024-09-06T14:24:55.448682+0000","last_fullsized":"2024-09-06T14:24:55.448682+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_clean_scrub_stamp":"2024-09-06T14:24:31.009248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-08T01:09:45.306197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"60'19","reported_seq":57,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.921962+0000","last_change":"2024-09-06T14:24:28.064397+0000","last_active":"2024-09-06T14:24:55.921962+0000","last_peered":"2024-09-06T14:24:55.921962+0000","last_clean":"2024-09-06T14:24:55.921962+0000","last_became_active":"2024-09-06T14:24:28.064064+0000","last_became_peered":"2024-09-06T14:24:28.064064+0000","last_unstale":"2024-09-06T14:24:55.921962+0000","last_undegraded":"2024-09-06T14:24:55.921962+0000","last_fullsized":"2024-09-06T14:24:55.921962+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_clean_scrub_stamp":"2024-09-06T14:24:26.984416+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T16:29:03.138477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":30,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.448222+0000","last_change":"2024-09-06T14:24:25.996040+0000","last_active":"2024-09-06T14:24:55.448222+0000","last_peered":"2024-09-06T14:24:55.448222+0000","last_clean":"2024-09-06T14:24:55.448222+0000","last_became_active":"2024-09-06T14:24:25.995769+0000","last_became_peered":"2024-09-06T14:24:25.995769+0000","last_unstale":"2024-09-06T14:24:55.448222+0000","last_undegraded":"2024-09-06T14:24:55.448222+0000","last_fullsized":"2024-09-06T14:24:55.448222+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_clean_scrub_stamp":"2024-09-06T14:24:24.970251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T20:43:32.771519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":22,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.841955+0000","last_change":"2024-09-06T14:24:30.024892+0000","last_active":"2024-09-06T14:24:55.841955+0000","last_peered":"2024-09-06T14:24:55.841955+0000","last_clean":"2024-09-06T14:24:55.841955+0000","last_became_active":"2024-09-06T14:24:30.024549+0000","last_became_peered":"2024-09-06T14:24:30.024549+0000","last_unstale":"2024-09-06T14:24:55.841955+0000","last_undegraded":"2024-09-06T14:24:55.841955+0000","last_fullsized":"2024-09-06T14:24:55.841955+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_clean_scrub_stamp":"2024-09-06T14:24:28.992633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T23:11:05.742408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":18,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.447142+0000","last_change":"2024-09-06T14:24:32.039789+0000","last_active":"2024-09-06T14:24:55.447142+0000","last_peered":"2024-09-06T14:24:55.447142+0000","last_clean":"2024-09-06T14:24:55.447142+0000","last_became_active":"2024-09-06T14:24:32.039582+0000","last_became_peered":"2024-09-06T14:24:32.039582+0000","last_unstale":"2024-09-06T14:24:55.447142+0000","last_undegraded":"2024-09-06T14:24:55.447142+0000","last_fullsized":"2024-09-06T14:24:55.447142+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_clean_scrub_stamp":"2024-09-06T14:24:31.009248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T15:15:24.485063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"60'12","reported_seq":49,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.921931+0000","last_change":"2024-09-06T14:24:28.013600+0000","last_active":"2024-09-06T14:24:55.921931+0000","last_peered":"2024-09-06T14:24:55.921931+0000","last_clean":"2024-09-06T14:24:55.921931+0000","last_became_active":"2024-09-06T14:24:28.013228+0000","last_became_peered":"2024-09-06T14:24:28.013228+0000","last_unstale":"2024-09-06T14:24:55.921931+0000","last_undegraded":"2024-09-06T14:24:55.921931+0000","last_fullsized":"2024-09-06T14:24:55.921931+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_clean_scrub_stamp":"2024-09-06T14:24:26.984416+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T16:55:36.416197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":30,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.448774+0000","last_change":"2024-09-06T14:24:25.999147+0000","last_active":"2024-09-06T14:24:55.448774+0000","last_peered":"2024-09-06T14:24:55.448774+0000","last_clean":"2024-09-06T14:24:55.448774+0000","last_became_active":"2024-09-06T14:24:25.998793+0000","last_became_peered":"2024-09-06T14:24:25.998793+0000","last_unstale":"2024-09-06T14:24:55.448774+0000","last_undegraded":"2024-09-06T14:24:55.448774+0000","last_fullsized":"2024-09-06T14:24:55.448774+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_clean_scrub_stamp":"2024-09-06T14:24:24.970251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T15:58:20.551192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":22,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.842074+0000","last_change":"2024-09-06T14:24:30.024147+0000","last_active":"2024-09-06T14:24:55.842074+0000","last_peered":"2024-09-06T14:24:55.842074+0000","last_clean":"2024-09-06T14:24:55.842074+0000","last_became_active":"2024-09-06T14:24:30.023785+0000","last_became_peered":"2024-09-06T14:24:30.023785+0000","last_unstale":"2024-09-06T14:24:55.842074+0000","last_undegraded":"2024-09-06T14:24:55.842074+0000","last_fullsized":"2024-09-06T14:24:55.842074+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_clean_scrub_stamp":"2024-09-06T14:24:28.992633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T17:19:53.871358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":18,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.449826+0000","last_change":"2024-09-06T14:24:32.039389+0000","last_active":"2024-09-06T14:24:55.449826+0000","last_peered":"2024-09-06T14:24:55.449826+0000","last_clean":"2024-09-06T14:24:55.449826+0000","last_became_active":"2024-09-06T14:24:32.038270+0000","last_became_peered":"2024-09-06T14:24:32.038270+0000","last_unstale":"2024-09-06T14:24:55.449826+0000","last_undegraded":"2024-09-06T14:24:55.449826+0000","last_fullsized":"2024-09-06T14:24:55.449826+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_clean_scrub_stamp":"2024-09-06T14:24:31.009248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T15:52:23.975701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"60'15","reported_seq":51,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.785619+0000","last_change":"2024-09-06T14:24:28.062462+0000","last_active":"2024-09-06T14:24:55.785619+0000","last_peered":"2024-09-06T14:24:55.785619+0000","last_clean":"2024-09-06T14:24:55.785619+0000","last_became_active":"2024-09-06T14:24:28.062144+0000","last_became_peered":"2024-09-06T14:24:28.062144+0000","last_unstale":"2024-09-06T14:24:55.785619+0000","last_undegraded":"2024-09-06T14:24:55.785619+0000","last_fullsized":"2024-09-06T14:24:55.785619+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_clean_scrub_stamp":"2024-09-06T14:24:26.984416+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-08T02:12:19.975625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"53'2","reported_seq":42,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.448443+0000","last_change":"2024-09-06T14:24:26.001845+0000","last_active":"2024-09-06T14:24:55.448443+0000","last_peered":"2024-09-06T14:24:55.448443+0000","last_clean":"2024-09-06T14:24:55.448443+0000","last_became_active":"2024-09-06T14:24:26.001579+0000","last_became_peered":"2024-09-06T14:24:26.001579+0000","last_unstale":"2024-09-06T14:24:55.448443+0000","last_undegraded":"2024-09-06T14:24:55.448443+0000","last_fullsized":"2024-09-06T14:24:55.448443+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_clean_scrub_stamp":"2024-09-06T14:24:24.970251+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-08T01:53:37.000913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"60'8","reported_seq":40,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:25:06.056621+0000","last_change":"2024-09-06T14:24:30.019173+0000","last_active":"2024-09-06T14:25:06.056621+0000","last_peered":"2024-09-06T14:25:06.056621+0000","last_clean":"2024-09-06T14:25:06.056621+0000","last_became_active":"2024-09-06T14:24:30.018792+0000","last_became_peered":"2024-09-06T14:24:30.018792+0000","last_unstale":"2024-09-06T14:25:06.056621+0000","last_undegraded":"2024-09-06T14:25:06.056621+0000","last_fullsized":"2024-09-06T14:25:06.056621+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_clean_scrub_stamp":"2024-09-06T14:24:28.992633+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T19:46:37.295207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":18,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.785728+0000","last_change":"2024-09-06T14:24:32.032978+0000","last_active":"2024-09-06T14:24:55.785728+0000","last_peered":"2024-09-06T14:24:55.785728+0000","last_clean":"2024-09-06T14:24:55.785728+0000","last_became_active":"2024-09-06T14:24:32.032701+0000","last_became_peered":"2024-09-06T14:24:32.032701+0000","last_unstale":"2024-09-06T14:24:55.785728+0000","last_undegraded":"2024-09-06T14:24:55.785728+0000","last_fullsized":"2024-09-06T14:24:55.785728+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_clean_scrub_stamp":"2024-09-06T14:24:31.009248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T16:38:24.537554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":30,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.921719+0000","last_change":"2024-09-06T14:24:25.994783+0000","last_active":"2024-09-06T14:24:55.921719+0000","last_peered":"2024-09-06T14:24:55.921719+0000","last_clean":"2024-09-06T14:24:55.921719+0000","last_became_active":"2024-09-06T14:24:25.994514+0000","last_became_peered":"2024-09-06T14:24:25.994514+0000","last_unstale":"2024-09-06T14:24:55.921719+0000","last_undegraded":"2024-09-06T14:24:55.921719+0000","last_fullsized":"2024-09-06T14:24:55.921719+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_clean_scrub_stamp":"2024-09-06T14:24:24.970251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T17:55:21.826188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"60'6","reported_seq":35,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.451831+0000","last_change":"2024-09-06T14:24:28.065250+0000","last_active":"2024-09-06T14:24:55.451831+0000","last_peered":"2024-09-06T14:24:55.451831+0000","last_clean":"2024-09-06T14:24:55.451831+0000","last_became_active":"2024-09-06T14:24:28.064819+0000","last_became_peered":"2024-09-06T14:24:28.064819+0000","last_unstale":"2024-09-06T14:24:55.451831+0000","last_undegraded":"2024-09-06T14:24:55.451831+0000","last_fullsized":"2024-09-06T14:24:55.451831+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_clean_scrub_stamp":"2024-09-06T14:24:26.984416+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-08T01:01:44.441868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":22,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.786130+0000","last_change":"2024-09-06T14:24:30.018751+0000","last_active":"2024-09-06T14:24:55.786130+0000","last_peered":"2024-09-06T14:24:55.786130+0000","last_clean":"2024-09-06T14:24:55.786130+0000","last_became_active":"2024-09-06T14:24:30.018294+0000","last_became_peered":"2024-09-06T14:24:30.018294+0000","last_unstale":"2024-09-06T14:24:55.786130+0000","last_undegraded":"2024-09-06T14:24:55.786130+0000","last_fullsized":"2024-09-06T14:24:55.786130+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_clean_scrub_stamp":"2024-09-06T14:24:28.992633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T19:52:16.089427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":18,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.447624+0000","last_change":"2024-09-06T14:24:32.031674+0000","last_active":"2024-09-06T14:24:55.447624+0000","last_peered":"2024-09-06T14:24:55.447624+0000","last_clean":"2024-09-06T14:24:55.447624+0000","last_became_active":"2024-09-06T14:24:32.030598+0000","last_became_peered":"2024-09-06T14:24:32.030598+0000","last_unstale":"2024-09-06T14:24:55.447624+0000","last_undegraded":"2024-09-06T14:24:55.447624+0000","last_fullsized":"2024-09-06T14:24:55.447624+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_clean_scrub_stamp":"2024-09-06T14:24:31.009248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T23:16:38.282134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"60'9","reported_seq":42,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.451230+0000","last_change":"2024-09-06T14:24:28.064252+0000","last_active":"2024-09-06T14:24:55.451230+0000","last_peered":"2024-09-06T14:24:55.451230+0000","last_clean":"2024-09-06T14:24:55.451230+0000","last_became_active":"2024-09-06T14:24:28.063935+0000","last_became_peered":"2024-09-06T14:24:28.063935+0000","last_unstale":"2024-09-06T14:24:55.451230+0000","last_undegraded":"2024-09-06T14:24:55.451230+0000","last_fullsized":"2024-09-06T14:24:55.451230+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_clean_scrub_stamp":"2024-09-06T14:24:26.984416+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T18:19:35.669936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":30,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.447253+0000","last_change":"2024-09-06T14:24:25.997064+0000","last_active":"2024-09-06T14:24:55.447253+0000","last_peered":"2024-09-06T14:24:55.447253+0000","last_clean":"2024-09-06T14:24:55.447253+0000","last_became_active":"2024-09-06T14:24:25.996789+0000","last_became_peered":"2024-09-06T14:24:25.996789+0000","last_unstale":"2024-09-06T14:24:55.447253+0000","last_undegraded":"2024-09-06T14:24:55.447253+0000","last_fullsized":"2024-09-06T14:24:55.447253+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_clean_scrub_stamp":"2024-09-06T14:24:24.970251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T14:27:54.007057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":22,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.451692+0000","last_change":"2024-09-06T14:24:30.017003+0000","last_active":"2024-09-06T14:24:55.451692+0000","last_peered":"2024-09-06T14:24:55.451692+0000","last_clean":"2024-09-06T14:24:55.451692+0000","last_became_active":"2024-09-06T14:24:30.016696+0000","last_became_peered":"2024-09-06T14:24:30.016696+0000","last_unstale":"2024-09-06T14:24:55.451692+0000","last_undegraded":"2024-09-06T14:24:55.451692+0000","last_fullsized":"2024-09-06T14:24:55.451692+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_clean_scrub_stamp":"2024-09-06T14:24:28.992633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-08T01:02:32.539435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":18,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.842289+0000","last_change":"2024-09-06T14:24:32.033331+0000","last_active":"2024-09-06T14:24:55.842289+0000","last_peered":"2024-09-06T14:24:55.842289+0000","last_clean":"2024-09-06T14:24:55.842289+0000","last_became_active":"2024-09-06T14:24:32.033040+0000","last_became_peered":"2024-09-06T14:24:32.033040+0000","last_unstale":"2024-09-06T14:24:55.842289+0000","last_undegraded":"2024-09-06T14:24:55.842289+0000","last_fullsized":"2024-09-06T14:24:55.842289+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_clean_scrub_stamp":"2024-09-06T14:24:31.009248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T18:40:02.034547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"60'9","reported_seq":42,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.786213+0000","last_change":"2024-09-06T14:24:28.064398+0000","last_active":"2024-09-06T14:24:55.786213+0000","last_peered":"2024-09-06T14:24:55.786213+0000","last_clean":"2024-09-06T14:24:55.786213+0000","last_became_active":"2024-09-06T14:24:28.064115+0000","last_became_peered":"2024-09-06T14:24:28.064115+0000","last_unstale":"2024-09-06T14:24:55.786213+0000","last_undegraded":"2024-09-06T14:24:55.786213+0000","last_fullsized":"2024-09-06T14:24:55.786213+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_clean_scrub_stamp":"2024-09-06T14:24:26.984416+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-08T01:33:05.303141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":30,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.449067+0000","last_change":"2024-09-06T14:24:26.001890+0000","last_active":"2024-09-06T14:24:55.449067+0000","last_peered":"2024-09-06T14:24:55.449067+0000","last_clean":"2024-09-06T14:24:55.449067+0000","last_became_active":"2024-09-06T14:24:26.001120+0000","last_became_peered":"2024-09-06T14:24:26.001120+0000","last_unstale":"2024-09-06T14:24:55.449067+0000","last_undegraded":"2024-09-06T14:24:55.449067+0000","last_fullsized":"2024-09-06T14:24:55.449067+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_clean_scrub_stamp":"2024-09-06T14:24:24.970251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T18:31:07.879304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"60'8","reported_seq":40,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:25:06.056622+0000","last_change":"2024-09-06T14:24:30.020387+0000","last_active":"2024-09-06T14:25:06.056622+0000","last_peered":"2024-09-06T14:25:06.056622+0000","last_clean":"2024-09-06T14:25:06.056622+0000","last_became_active":"2024-09-06T14:24:30.020097+0000","last_became_peered":"2024-09-06T14:24:30.020097+0000","last_unstale":"2024-09-06T14:25:06.056622+0000","last_undegraded":"2024-09-06T14:25:06.056622+0000","last_fullsized":"2024-09-06T14:25:06.056622+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_clean_scrub_stamp":"2024-09-06T14:24:28.992633+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-08T01:50:20.984268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":18,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.922349+0000","last_change":"2024-09-06T14:24:32.034910+0000","last_active":"2024-09-06T14:24:55.922349+0000","last_peered":"2024-09-06T14:24:55.922349+0000","last_clean":"2024-09-06T14:24:55.922349+0000","last_became_active":"2024-09-06T14:24:32.034628+0000","last_became_peered":"2024-09-06T14:24:32.034628+0000","last_unstale":"2024-09-06T14:24:55.922349+0000","last_undegraded":"2024-09-06T14:24:55.922349+0000","last_fullsized":"2024-09-06T14:24:55.922349+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_clean_scrub_stamp":"2024-09-06T14:24:31.009248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-08T01:59:51.503411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"60'10","reported_seq":41,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.450411+0000","last_change":"2024-09-06T14:24:28.062417+0000","last_active":"2024-09-06T14:24:55.450411+0000","last_peered":"2024-09-06T14:24:55.450411+0000","last_clean":"2024-09-06T14:24:55.450411+0000","last_became_active":"2024-09-06T14:24:28.062108+0000","last_became_peered":"2024-09-06T14:24:28.062108+0000","last_unstale":"2024-09-06T14:24:55.450411+0000","last_undegraded":"2024-09-06T14:24:55.450411+0000","last_fullsized":"2024-09-06T14:24:55.450411+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_clean_scrub_stamp":"2024-09-06T14:24:26.984416+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T21:52:05.503683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":30,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.447607+0000","last_change":"2024-09-06T14:24:25.997813+0000","last_active":"2024-09-06T14:24:55.447607+0000","last_peered":"2024-09-06T14:24:55.447607+0000","last_clean":"2024-09-06T14:24:55.447607+0000","last_became_active":"2024-09-06T14:24:25.997546+0000","last_became_peered":"2024-09-06T14:24:25.997546+0000","last_unstale":"2024-09-06T14:24:55.447607+0000","last_undegraded":"2024-09-06T14:24:55.447607+0000","last_fullsized":"2024-09-06T14:24:55.447607+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_clean_scrub_stamp":"2024-09-06T14:24:24.970251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T20:13:00.222134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"60'8","reported_seq":40,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:25:06.056657+0000","last_change":"2024-09-06T14:24:30.018657+0000","last_active":"2024-09-06T14:25:06.056657+0000","last_peered":"2024-09-06T14:25:06.056657+0000","last_clean":"2024-09-06T14:25:06.056657+0000","last_became_active":"2024-09-06T14:24:30.018356+0000","last_became_peered":"2024-09-06T14:24:30.018356+0000","last_unstale":"2024-09-06T14:25:06.056657+0000","last_undegraded":"2024-09-06T14:25:06.056657+0000","last_fullsized":"2024-09-06T14:25:06.056657+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_clean_scrub_stamp":"2024-09-06T14:24:28.992633+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-08T00:47:58.743872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":18,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.450745+0000","last_change":"2024-09-06T14:24:32.032991+0000","last_active":"2024-09-06T14:24:55.450745+0000","last_peered":"2024-09-06T14:24:55.450745+0000","last_clean":"2024-09-06T14:24:55.450745+0000","last_became_active":"2024-09-06T14:24:32.032691+0000","last_became_peered":"2024-09-06T14:24:32.032691+0000","last_unstale":"2024-09-06T14:24:55.450745+0000","last_undegraded":"2024-09-06T14:24:55.450745+0000","last_fullsized":"2024-09-06T14:24:55.450745+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_clean_scrub_stamp":"2024-09-06T14:24:31.009248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T22:17:45.055713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"60'11","reported_seq":45,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.922163+0000","last_change":"2024-09-06T14:24:28.004573+0000","last_active":"2024-09-06T14:24:55.922163+0000","last_peered":"2024-09-06T14:24:55.922163+0000","last_clean":"2024-09-06T14:24:55.922163+0000","last_became_active":"2024-09-06T14:24:28.004104+0000","last_became_peered":"2024-09-06T14:24:28.004104+0000","last_unstale":"2024-09-06T14:24:55.922163+0000","last_undegraded":"2024-09-06T14:24:55.922163+0000","last_fullsized":"2024-09-06T14:24:55.922163+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_clean_scrub_stamp":"2024-09-06T14:24:26.984416+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T15:30:21.462868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":30,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.922850+0000","last_change":"2024-09-06T14:24:26.001351+0000","last_active":"2024-09-06T14:24:55.922850+0000","last_peered":"2024-09-06T14:24:55.922850+0000","last_clean":"2024-09-06T14:24:55.922850+0000","last_became_active":"2024-09-06T14:24:26.000152+0000","last_became_peered":"2024-09-06T14:24:26.000152+0000","last_unstale":"2024-09-06T14:24:55.922850+0000","last_undegraded":"2024-09-06T14:24:55.922850+0000","last_fullsized":"2024-09-06T14:24:55.922850+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_clean_scrub_stamp":"2024-09-06T14:24:24.970251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-08T00:27:15.911914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":22,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.922627+0000","last_change":"2024-09-06T14:24:30.019277+0000","last_active":"2024-09-06T14:24:55.922627+0000","last_peered":"2024-09-06T14:24:55.922627+0000","last_clean":"2024-09-06T14:24:55.922627+0000","last_became_active":"2024-09-06T14:24:30.018902+0000","last_became_peered":"2024-09-06T14:24:30.018902+0000","last_unstale":"2024-09-06T14:24:55.922627+0000","last_undegraded":"2024-09-06T14:24:55.922627+0000","last_fullsized":"2024-09-06T14:24:55.922627+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_clean_scrub_stamp":"2024-09-06T14:24:28.992633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T19:36:50.484871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":18,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.449549+0000","last_change":"2024-09-06T14:24:32.037128+0000","last_active":"2024-09-06T14:24:55.449549+0000","last_peered":"2024-09-06T14:24:55.449549+0000","last_clean":"2024-09-06T14:24:55.449549+0000","last_became_active":"2024-09-06T14:24:32.036224+0000","last_became_peered":"2024-09-06T14:24:32.036224+0000","last_unstale":"2024-09-06T14:24:55.449549+0000","last_undegraded":"2024-09-06T14:24:55.449549+0000","last_fullsized":"2024-09-06T14:24:55.449549+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_clean_scrub_stamp":"2024-09-06T14:24:31.009248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-08T00:02:25.670231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"60'9","reported_seq":42,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.922923+0000","last_change":"2024-09-06T14:24:28.009721+0000","last_active":"2024-09-06T14:24:55.922923+0000","last_peered":"2024-09-06T14:24:55.922923+0000","last_clean":"2024-09-06T14:24:55.922923+0000","last_became_active":"2024-09-06T14:24:28.009420+0000","last_became_peered":"2024-09-06T14:24:28.009420+0000","last_unstale":"2024-09-06T14:24:55.922923+0000","last_undegraded":"2024-09-06T14:24:55.922923+0000","last_fullsized":"2024-09-06T14:24:55.922923+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_clean_scrub_stamp":"2024-09-06T14:24:26.984416+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-08T01:05:42.106662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":30,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.448076+0000","last_change":"2024-09-06T14:24:25.998847+0000","last_active":"2024-09-06T14:24:55.448076+0000","last_peered":"2024-09-06T14:24:55.448076+0000","last_clean":"2024-09-06T14:24:55.448076+0000","last_became_active":"2024-09-06T14:24:25.998566+0000","last_became_peered":"2024-09-06T14:24:25.998566+0000","last_unstale":"2024-09-06T14:24:55.448076+0000","last_undegraded":"2024-09-06T14:24:55.448076+0000","last_fullsized":"2024-09-06T14:24:55.448076+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_clean_scrub_stamp":"2024-09-06T14:24:24.970251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T20:25:07.723740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":22,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.449936+0000","last_change":"2024-09-06T14:24:30.015643+0000","last_active":"2024-09-06T14:24:55.449936+0000","last_peered":"2024-09-06T14:24:55.449936+0000","last_clean":"2024-09-06T14:24:55.449936+0000","last_became_active":"2024-09-06T14:24:30.015322+0000","last_became_peered":"2024-09-06T14:24:30.015322+0000","last_unstale":"2024-09-06T14:24:55.449936+0000","last_undegraded":"2024-09-06T14:24:55.449936+0000","last_fullsized":"2024-09-06T14:24:55.449936+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_clean_scrub_stamp":"2024-09-06T14:24:28.992633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T16:13:02.441998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":18,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.448884+0000","last_change":"2024-09-06T14:24:32.029053+0000","last_active":"2024-09-06T14:24:55.448884+0000","last_peered":"2024-09-06T14:24:55.448884+0000","last_clean":"2024-09-06T14:24:55.448884+0000","last_became_active":"2024-09-06T14:24:32.028702+0000","last_became_peered":"2024-09-06T14:24:32.028702+0000","last_unstale":"2024-09-06T14:24:55.448884+0000","last_undegraded":"2024-09-06T14:24:55.448884+0000","last_fullsized":"2024-09-06T14:24:55.448884+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_clean_scrub_stamp":"2024-09-06T14:24:31.009248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T21:30:59.288934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"60'11","reported_seq":45,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.449720+0000","last_change":"2024-09-06T14:24:28.062589+0000","last_active":"2024-09-06T14:24:55.449720+0000","last_peered":"2024-09-06T14:24:55.449720+0000","last_clean":"2024-09-06T14:24:55.449720+0000","last_became_active":"2024-09-06T14:24:28.062290+0000","last_became_peered":"2024-09-06T14:24:28.062290+0000","last_unstale":"2024-09-06T14:24:55.449720+0000","last_undegraded":"2024-09-06T14:24:55.449720+0000","last_fullsized":"2024-09-06T14:24:55.449720+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_clean_scrub_stamp":"2024-09-06T14:24:26.984416+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T21:26:53.134114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":30,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.786196+0000","last_change":"2024-09-06T14:24:25.998515+0000","last_active":"2024-09-06T14:24:55.786196+0000","last_peered":"2024-09-06T14:24:55.786196+0000","last_clean":"2024-09-06T14:24:55.786196+0000","last_became_active":"2024-09-06T14:24:25.998174+0000","last_became_peered":"2024-09-06T14:24:25.998174+0000","last_unstale":"2024-09-06T14:24:55.786196+0000","last_undegraded":"2024-09-06T14:24:55.786196+0000","last_fullsized":"2024-09-06T14:24:55.786196+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_clean_scrub_stamp":"2024-09-06T14:24:24.970251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T21:18:35.673089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":22,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.446933+0000","last_change":"2024-09-06T14:24:30.015894+0000","last_active":"2024-09-06T14:24:55.446933+0000","last_peered":"2024-09-06T14:24:55.446933+0000","last_clean":"2024-09-06T14:24:55.446933+0000","last_became_active":"2024-09-06T14:24:30.015624+0000","last_became_peered":"2024-09-06T14:24:30.015624+0000","last_unstale":"2024-09-06T14:24:55.446933+0000","last_undegraded":"2024-09-06T14:24:55.446933+0000","last_fullsized":"2024-09-06T14:24:55.446933+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_clean_scrub_stamp":"2024-09-06T14:24:28.992633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T22:36:13.838324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":18,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.450088+0000","last_change":"2024-09-06T14:24:32.038464+0000","last_active":"2024-09-06T14:24:55.450088+0000","last_peered":"2024-09-06T14:24:55.450088+0000","last_clean":"2024-09-06T14:24:55.450088+0000","last_became_active":"2024-09-06T14:24:32.037605+0000","last_became_peered":"2024-09-06T14:24:32.037605+0000","last_unstale":"2024-09-06T14:24:55.450088+0000","last_undegraded":"2024-09-06T14:24:55.450088+0000","last_fullsized":"2024-09-06T14:24:55.450088+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_clean_scrub_stamp":"2024-09-06T14:24:31.009248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T22:48:10.569670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"60'4","reported_seq":32,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.449718+0000","last_change":"2024-09-06T14:24:28.015730+0000","last_active":"2024-09-06T14:24:55.449718+0000","last_peered":"2024-09-06T14:24:55.449718+0000","last_clean":"2024-09-06T14:24:55.449718+0000","last_became_active":"2024-09-06T14:24:28.015423+0000","last_became_peered":"2024-09-06T14:24:28.015423+0000","last_unstale":"2024-09-06T14:24:55.449718+0000","last_undegraded":"2024-09-06T14:24:55.449718+0000","last_fullsized":"2024-09-06T14:24:55.449718+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_clean_scrub_stamp":"2024-09-06T14:24:26.984416+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T14:54:45.334421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"53'1","reported_seq":36,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.449856+0000","last_change":"2024-09-06T14:24:26.002065+0000","last_active":"2024-09-06T14:24:55.449856+0000","last_peered":"2024-09-06T14:24:55.449856+0000","last_clean":"2024-09-06T14:24:55.449856+0000","last_became_active":"2024-09-06T14:24:25.999543+0000","last_became_peered":"2024-09-06T14:24:25.999543+0000","last_unstale":"2024-09-06T14:24:55.449856+0000","last_undegraded":"2024-09-06T14:24:55.449856+0000","last_fullsized":"2024-09-06T14:24:55.449856+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_clean_scrub_stamp":"2024-09-06T14:24:24.970251+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T20:12:56.043359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":22,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.921711+0000","last_change":"2024-09-06T14:24:30.014764+0000","last_active":"2024-09-06T14:24:55.921711+0000","last_peered":"2024-09-06T14:24:55.921711+0000","last_clean":"2024-09-06T14:24:55.921711+0000","last_became_active":"2024-09-06T14:24:30.014404+0000","last_became_peered":"2024-09-06T14:24:30.014404+0000","last_unstale":"2024-09-06T14:24:55.921711+0000","last_undegraded":"2024-09-06T14:24:55.921711+0000","last_fullsized":"2024-09-06T14:24:55.921711+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_clean_scrub_stamp":"2024-09-06T14:24:28.992633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T17:19:26.404116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":18,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.447142+0000","last_change":"2024-09-06T14:24:32.031767+0000","last_active":"2024-09-06T14:24:55.447142+0000","last_peered":"2024-09-06T14:24:55.447142+0000","last_clean":"2024-09-06T14:24:55.447142+0000","last_became_active":"2024-09-06T14:24:32.031430+0000","last_became_peered":"2024-09-06T14:24:32.031430+0000","last_unstale":"2024-09-06T14:24:55.447142+0000","last_undegraded":"2024-09-06T14:24:55.447142+0000","last_fullsized":"2024-09-06T14:24:55.447142+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_clean_scrub_stamp":"2024-09-06T14:24:31.009248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T19:35:08.072702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":18,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.922982+0000","last_change":"2024-09-06T14:24:32.033967+0000","last_active":"2024-09-06T14:24:55.922982+0000","last_peered":"2024-09-06T14:24:55.922982+0000","last_clean":"2024-09-06T14:24:55.922982+0000","last_became_active":"2024-09-06T14:24:32.033288+0000","last_became_peered":"2024-09-06T14:24:32.033288+0000","last_unstale":"2024-09-06T14:24:55.922982+0000","last_undegraded":"2024-09-06T14:24:55.922982+0000","last_fullsized":"2024-09-06T14:24:55.922982+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_clean_scrub_stamp":"2024-09-06T14:24:31.009248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T14:26:15.491944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":30,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.450772+0000","last_change":"2024-09-06T14:24:26.006072+0000","last_active":"2024-09-06T14:24:55.450772+0000","last_peered":"2024-09-06T14:24:55.450772+0000","last_clean":"2024-09-06T14:24:55.450772+0000","last_became_active":"2024-09-06T14:24:26.005840+0000","last_became_peered":"2024-09-06T14:24:26.005840+0000","last_unstale":"2024-09-06T14:24:55.450772+0000","last_undegraded":"2024-09-06T14:24:55.450772+0000","last_fullsized":"2024-09-06T14:24:55.450772+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_clean_scrub_stamp":"2024-09-06T14:24:24.970251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T19:45:19.119033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"60'11","reported_seq":45,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.922633+0000","last_change":"2024-09-06T14:24:28.008444+0000","last_active":"2024-09-06T14:24:55.922633+0000","last_peered":"2024-09-06T14:24:55.922633+0000","last_clean":"2024-09-06T14:24:55.922633+0000","last_became_active":"2024-09-06T14:24:28.008126+0000","last_became_peered":"2024-09-06T14:24:28.008126+0000","last_unstale":"2024-09-06T14:24:55.922633+0000","last_undegraded":"2024-09-06T14:24:55.922633+0000","last_fullsized":"2024-09-06T14:24:55.922633+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_clean_scrub_stamp":"2024-09-06T14:24:26.984416+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T22:28:51.014533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":22,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.450892+0000","last_change":"2024-09-06T14:24:30.017624+0000","last_active":"2024-09-06T14:24:55.450892+0000","last_peered":"2024-09-06T14:24:55.450892+0000","last_clean":"2024-09-06T14:24:55.450892+0000","last_became_active":"2024-09-06T14:24:30.017352+0000","last_became_peered":"2024-09-06T14:24:30.017352+0000","last_unstale":"2024-09-06T14:24:55.450892+0000","last_undegraded":"2024-09-06T14:24:55.450892+0000","last_fullsized":"2024-09-06T14:24:55.450892+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_clean_scrub_stamp":"2024-09-06T14:24:28.992633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T16:03:39.031274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":40,"num_read_kb":35,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":48,"seq":206158430221,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32380,"kb_used_data":1192,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712772,"statfs":{"total":95995035648,"available":95961878528,"internally_reserved":0,"allocated":1220608,"data_stored":822957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626450,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28200,"kb_used_data":1176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716952,"statfs":{"total":95995035648,"available":95966158848,"internally_reserved":0,"allocated":1204224,"data_stored":822308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822679,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31788,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713364,"statfs":{"total":95995035648,"available":95962484736,"internally_reserved":0,"allocated":614400,"data_stored":232856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849018908,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27656,"kb_used_data":628,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717496,"statfs":{"total":95995035648,"available":95966715904,"internally_reserved":0,"allocated":643072,"data_stored":232558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182434,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31820,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713332,"statfs":{"total":95995035648,"available":95962451968,"internally_reserved":0,"allocated":647168,"data_stored":233388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411366,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31780,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713372,"statfs":{"total":95995035648,"available":95962492928,"internally_reserved":0,"allocated":606208,"data_stored":231665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574891,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27692,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717460,"statfs":{"total":95995035648,"available":95966679040,"internally_reserved":0,"allocated":679936,"data_stored":233738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738417,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28224,"kb_used_data":1200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716928,"statfs":{"total":95995035648,"available":95966134272,"internally_reserved":0,"allocated":1228800,"data_stored":823659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-06T14:25:12.962 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph pg dump --format=json 2024-09-06T14:25:13.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:12 smithi142 ceph-mon[29958]: pgmap v11: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-06T14:25:13.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:12 smithi142 ceph-mon[29958]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:13.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:12 smithi142 ceph-mon[29958]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:13.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:12 smithi142 ceph-mon[29958]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-06T14:25:13.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:12 smithi142 ceph-mon[29958]: from='mgr.14601 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-06T14:25:13.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:12 smithi142 ceph-mon[29958]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-06T14:25:13.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:12 smithi142 ceph-mon[29958]: from='mgr.14601 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-06T14:25:13.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:12 smithi142 ceph-mon[29958]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-06T14:25:13.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:12 smithi142 ceph-mon[29958]: from='mgr.14601 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-06T14:25:13.210 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:12 smithi142 ceph-mon[29958]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-06T14:25:13.210 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:12 smithi142 ceph-mon[29958]: from='mgr.14601 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-06T14:25:13.210 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:12 smithi142 ceph-mon[29958]: Adjusting osd_memory_target on smithi142 to 3760M 2024-09-06T14:25:13.210 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:12 smithi142 ceph-mon[29958]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:13.210 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:12 smithi142 ceph-mon[29958]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:25:13.210 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:12 smithi142 ceph-mon[29958]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-06T14:25:13.210 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:12 smithi142 ceph-mon[29958]: Updating smithi142:/etc/ceph/ceph.conf 2024-09-06T14:25:13.210 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:12 smithi142 ceph-mon[29958]: Updating smithi162:/etc/ceph/ceph.conf 2024-09-06T14:25:13.210 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:12 smithi142 ceph-mon[29958]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:13.210 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:12 smithi142 ceph-mon[29958]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:13.210 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:12 smithi142 ceph-mon[29958]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:13.210 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:12 smithi142 ceph-mon[29958]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:13.210 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:12 smithi142 ceph-mon[29958]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:13.211 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:12 smithi142 ceph-mon[37372]: pgmap v11: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-06T14:25:13.211 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:12 smithi142 ceph-mon[37372]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:13.211 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:12 smithi142 ceph-mon[37372]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:13.211 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:12 smithi142 ceph-mon[37372]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-06T14:25:13.211 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:12 smithi142 ceph-mon[37372]: from='mgr.14601 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-06T14:25:13.211 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:12 smithi142 ceph-mon[37372]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-06T14:25:13.211 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:12 smithi142 ceph-mon[37372]: from='mgr.14601 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-06T14:25:13.211 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:12 smithi142 ceph-mon[37372]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-06T14:25:13.211 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:12 smithi142 ceph-mon[37372]: from='mgr.14601 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-06T14:25:13.211 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:12 smithi142 ceph-mon[37372]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-06T14:25:13.211 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:12 smithi142 ceph-mon[37372]: from='mgr.14601 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-06T14:25:13.212 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:12 smithi142 ceph-mon[37372]: Adjusting osd_memory_target on smithi142 to 3760M 2024-09-06T14:25:13.212 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:12 smithi142 ceph-mon[37372]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:13.212 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:12 smithi142 ceph-mon[37372]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:25:13.212 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:12 smithi142 ceph-mon[37372]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-06T14:25:13.212 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:12 smithi142 ceph-mon[37372]: Updating smithi142:/etc/ceph/ceph.conf 2024-09-06T14:25:13.212 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:12 smithi142 ceph-mon[37372]: Updating smithi162:/etc/ceph/ceph.conf 2024-09-06T14:25:13.212 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:12 smithi142 ceph-mon[37372]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:13.212 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:12 smithi142 ceph-mon[37372]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:13.212 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:12 smithi142 ceph-mon[37372]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:13.212 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:12 smithi142 ceph-mon[37372]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:13.212 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:12 smithi142 ceph-mon[37372]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:13.221 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:25:13.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:12 smithi162 ceph-mon[33787]: pgmap v11: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-06T14:25:13.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:12 smithi162 ceph-mon[33787]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:13.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:12 smithi162 ceph-mon[33787]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:13.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:12 smithi162 ceph-mon[33787]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-06T14:25:13.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:12 smithi162 ceph-mon[33787]: from='mgr.14601 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-06T14:25:13.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:12 smithi162 ceph-mon[33787]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-06T14:25:13.298 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:12 smithi162 ceph-mon[33787]: from='mgr.14601 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-06T14:25:13.298 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:12 smithi162 ceph-mon[33787]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-06T14:25:13.298 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:12 smithi162 ceph-mon[33787]: from='mgr.14601 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-06T14:25:13.298 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:12 smithi162 ceph-mon[33787]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-06T14:25:13.298 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:12 smithi162 ceph-mon[33787]: from='mgr.14601 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-06T14:25:13.298 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:12 smithi162 ceph-mon[33787]: Adjusting osd_memory_target on smithi142 to 3760M 2024-09-06T14:25:13.298 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:12 smithi162 ceph-mon[33787]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:13.298 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:12 smithi162 ceph-mon[33787]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:25:13.298 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:12 smithi162 ceph-mon[33787]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-06T14:25:13.298 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:12 smithi162 ceph-mon[33787]: Updating smithi142:/etc/ceph/ceph.conf 2024-09-06T14:25:13.298 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:12 smithi162 ceph-mon[33787]: Updating smithi162:/etc/ceph/ceph.conf 2024-09-06T14:25:13.298 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:12 smithi162 ceph-mon[33787]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:13.299 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:12 smithi162 ceph-mon[33787]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:13.299 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:12 smithi162 ceph-mon[33787]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:13.299 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:12 smithi162 ceph-mon[33787]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:13.299 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:12 smithi162 ceph-mon[33787]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:13.706 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:13.689+0000 7f61bd37e640 1 -- 172.21.15.142:0/436729737 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f61b8108510 msgr2=0x7f61b8108950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:25:13.706 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:13.689+0000 7f61bd37e640 1 --2- 172.21.15.142:0/436729737 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f61b8108510 0x7f61b8108950 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f61ac009900 tx=0x7f61ac02a410 comp rx=0 tx=0).stop 2024-09-06T14:25:13.706 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:13.689+0000 7f61bd37e640 1 -- 172.21.15.142:0/436729737 shutdown_connections 2024-09-06T14:25:13.707 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:13.689+0000 7f61bd37e640 1 --2- 172.21.15.142:0/436729737 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f61b8108e90 0x7f61b810f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:13.707 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:13.689+0000 7f61bd37e640 1 --2- 172.21.15.142:0/436729737 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f61b8108510 0x7f61b8108950 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:13.707 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:13.689+0000 7f61bd37e640 1 --2- 172.21.15.142:0/436729737 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f61b8102b00 0x7f61b8102ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:13.707 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:13.689+0000 7f61bd37e640 1 -- 172.21.15.142:0/436729737 >> 172.21.15.142:0/436729737 conn(0x7f61b80fe570 msgr2=0x7f61b8100990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:25:13.707 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:13.689+0000 7f61bd37e640 1 -- 172.21.15.142:0/436729737 shutdown_connections 2024-09-06T14:25:13.707 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:13.689+0000 7f61bd37e640 1 -- 172.21.15.142:0/436729737 wait complete. 2024-09-06T14:25:13.707 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:13.690+0000 7f61bd37e640 1 Processor -- start 2024-09-06T14:25:13.707 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:13.690+0000 7f61bd37e640 1 -- start start 2024-09-06T14:25:13.707 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:13.690+0000 7f61bd37e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f61b8102b00 0x7f61b81986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:25:13.707 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:13.690+0000 7f61bd37e640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f61b8108510 0x7f61b8198c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:25:13.707 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:13.690+0000 7f61bd37e640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f61b8108e90 0x7f61b819dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:25:13.707 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:13.690+0000 7f61bd37e640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f61b8112230 con 0x7f61b8108e90 2024-09-06T14:25:13.707 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:13.690+0000 7f61bd37e640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f61b81120b0 con 0x7f61b8102b00 2024-09-06T14:25:13.708 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:13.690+0000 7f61bd37e640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7f61b81123b0 con 0x7f61b8108510 2024-09-06T14:25:13.708 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:13.690+0000 7f61b67fc640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f61b8108510 0x7f61b8198c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:25:13.708 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:13.690+0000 7f61b77fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f61b8108e90 0x7f61b819dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:25:13.708 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:13.690+0000 7f61b67fc640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f61b8108510 0x7f61b8198c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:35684/0 (socket says 172.21.15.142:35684) 2024-09-06T14:25:13.708 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:13.690+0000 7f61b67fc640 1 -- 172.21.15.142:0/4175708916 learned_addr learned my addr 172.21.15.142:0/4175708916 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:25:13.708 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:13.690+0000 7f61b6ffd640 1 --2- 172.21.15.142:0/4175708916 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f61b8102b00 0x7f61b81986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:25:13.708 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:13.691+0000 7f61b67fc640 1 -- 172.21.15.142:0/4175708916 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f61b8102b00 msgr2=0x7f61b81986c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:25:13.708 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:13.691+0000 7f61b67fc640 1 --2- 172.21.15.142:0/4175708916 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f61b8102b00 0x7f61b81986c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:13.708 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:13.691+0000 7f61b67fc640 1 -- 172.21.15.142:0/4175708916 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f61b8108e90 msgr2=0x7f61b819dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:25:13.708 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:13.691+0000 7f61b67fc640 1 --2- 172.21.15.142:0/4175708916 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f61b8108e90 0x7f61b819dfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:13.708 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:13.691+0000 7f61b67fc640 1 -- 172.21.15.142:0/4175708916 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61b819e770 con 0x7f61b8108510 2024-09-06T14:25:13.708 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:13.691+0000 7f61b77fe640 1 --2- 172.21.15.142:0/4175708916 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f61b8108e90 0x7f61b819dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-06T14:25:13.708 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:13.691+0000 7f61b6ffd640 1 --2- 172.21.15.142:0/4175708916 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f61b8102b00 0x7f61b81986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-06T14:25:13.708 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:13.691+0000 7f61b67fc640 1 --2- 172.21.15.142:0/4175708916 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f61b8108510 0x7f61b8198c00 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f61ac005ab0 tx=0x7f61ac005080 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:25:13.709 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:13.691+0000 7f619ffff640 1 -- 172.21.15.142:0/4175708916 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61ac03f070 con 0x7f61b8108510 2024-09-06T14:25:13.709 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:13.691+0000 7f61bd37e640 1 -- 172.21.15.142:0/4175708916 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61b819ea00 con 0x7f61b8108510 2024-09-06T14:25:13.709 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:13.691+0000 7f619ffff640 1 -- 172.21.15.142:0/4175708916 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7f61ac003a70 con 0x7f61b8108510 2024-09-06T14:25:13.709 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:13.691+0000 7f619ffff640 1 -- 172.21.15.142:0/4175708916 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61ac035030 con 0x7f61b8108510 2024-09-06T14:25:13.709 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:13.691+0000 7f61bd37e640 1 -- 172.21.15.142:0/4175708916 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f61b81a5120 con 0x7f61b8108510 2024-09-06T14:25:13.709 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:13.692+0000 7f61bd37e640 1 -- 172.21.15.142:0/4175708916 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61b8102ee0 con 0x7f61b8108510 2024-09-06T14:25:13.709 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:13.693+0000 7f619ffff640 1 -- 172.21.15.142:0/4175708916 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 22) ==== 99844+0+0 (secure 0 0 0) 0x7f61ac042070 con 0x7f61b8108510 2024-09-06T14:25:13.709 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:13.693+0000 7f619ffff640 1 --2- 172.21.15.142:0/4175708916 >> v2:172.21.15.142:6800/2944545612 conn(0x7f61a0077620 0x7f61a0079ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:25:13.709 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:13.693+0000 7f61b6ffd640 1 --2- 172.21.15.142:0/4175708916 >> v2:172.21.15.142:6800/2944545612 conn(0x7f61a0077620 0x7f61a0079ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:25:13.709 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:13.693+0000 7f619ffff640 1 -- 172.21.15.142:0/4175708916 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(62..62 src has 1..62) ==== 6043+0+0 (secure 0 0 0) 0x7f61ac0ba100 con 0x7f61b8108510 2024-09-06T14:25:13.709 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:13.693+0000 7f61b6ffd640 1 --2- 172.21.15.142:0/4175708916 >> v2:172.21.15.142:6800/2944545612 conn(0x7f61a0077620 0x7f61a0079ae0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f61a80095f0 tx=0x7f61a8005f90 comp rx=0 tx=0).ready entity=mgr.14601 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:25:13.709 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:13.697+0000 7f619ffff640 1 -- 172.21.15.142:0/4175708916 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7f61ac087390 con 0x7f61b8108510 2024-09-06T14:25:13.846 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:13.850+0000 7f61bd37e640 1 -- 172.21.15.142:0/4175708916 --> v2:172.21.15.142:6800/2944545612 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f61b80632f0 con 0x7f61a0077620 2024-09-06T14:25:13.852 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:13.855+0000 7f619ffff640 1 -- 172.21.15.142:0/4175708916 <== mgr.14601 v2:172.21.15.142:6800/2944545612 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+346463 (secure 0 0 0) 0x7f61b80632f0 con 0x7f61a0077620 2024-09-06T14:25:13.855 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:25:13.857 INFO:teuthology.orchestra.run.smithi142.stderr:dumped all 2024-09-06T14:25:13.859 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:13.858+0000 7f61bd37e640 1 -- 172.21.15.142:0/4175708916 >> v2:172.21.15.142:6800/2944545612 conn(0x7f61a0077620 msgr2=0x7f61a0079ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:25:13.859 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:13.858+0000 7f61bd37e640 1 --2- 172.21.15.142:0/4175708916 >> v2:172.21.15.142:6800/2944545612 conn(0x7f61a0077620 0x7f61a0079ae0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f61a80095f0 tx=0x7f61a8005f90 comp rx=0 tx=0).stop 2024-09-06T14:25:13.859 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:13.858+0000 7f61bd37e640 1 -- 172.21.15.142:0/4175708916 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f61b8108510 msgr2=0x7f61b8198c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:25:13.859 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:13.858+0000 7f61bd37e640 1 --2- 172.21.15.142:0/4175708916 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f61b8108510 0x7f61b8198c00 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f61ac005ab0 tx=0x7f61ac005080 comp rx=0 tx=0).stop 2024-09-06T14:25:13.859 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:13.858+0000 7f61bd37e640 1 -- 172.21.15.142:0/4175708916 shutdown_connections 2024-09-06T14:25:13.859 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:13.858+0000 7f61bd37e640 1 --2- 172.21.15.142:0/4175708916 >> v2:172.21.15.142:6800/2944545612 conn(0x7f61a0077620 0x7f61a0079ae0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:13.859 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:13.858+0000 7f61bd37e640 1 --2- 172.21.15.142:0/4175708916 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f61b8108e90 0x7f61b819dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:13.859 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:13.858+0000 7f61bd37e640 1 --2- 172.21.15.142:0/4175708916 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f61b8108510 0x7f61b8198c00 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:13.860 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:13.858+0000 7f61bd37e640 1 --2- 172.21.15.142:0/4175708916 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f61b8102b00 0x7f61b81986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:13.860 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:13.858+0000 7f61bd37e640 1 -- 172.21.15.142:0/4175708916 >> 172.21.15.142:0/4175708916 conn(0x7f61b80fe570 msgr2=0x7f61b8100960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:25:13.860 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:13.858+0000 7f61bd37e640 1 -- 172.21.15.142:0/4175708916 shutdown_connections 2024-09-06T14:25:13.860 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:13.858+0000 7f61bd37e640 1 -- 172.21.15.142:0/4175708916 wait complete. 2024-09-06T14:25:14.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:13 smithi142 ceph-mon[29958]: from='client.14721 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-06T14:25:14.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:13 smithi142 ceph-mon[29958]: Updating smithi142:/var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/config/ceph.conf 2024-09-06T14:25:14.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:13 smithi142 ceph-mon[29958]: Updating smithi162:/var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/config/ceph.conf 2024-09-06T14:25:14.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:13 smithi142 ceph-mon[29958]: Updating smithi142:/etc/ceph/ceph.client.admin.keyring 2024-09-06T14:25:14.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:13 smithi142 ceph-mon[29958]: Updating smithi162:/etc/ceph/ceph.client.admin.keyring 2024-09-06T14:25:14.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:13 smithi142 ceph-mon[29958]: Updating smithi142:/var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/config/ceph.client.admin.keyring 2024-09-06T14:25:14.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:13 smithi142 ceph-mon[29958]: Updating smithi162:/var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/config/ceph.client.admin.keyring 2024-09-06T14:25:14.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:13 smithi142 ceph-mon[29958]: Deploying daemon alertmanager.a on smithi142 2024-09-06T14:25:14.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:13 smithi142 ceph-mon[37372]: from='client.14721 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-06T14:25:14.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:13 smithi142 ceph-mon[37372]: Updating smithi142:/var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/config/ceph.conf 2024-09-06T14:25:14.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:13 smithi142 ceph-mon[37372]: Updating smithi162:/var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/config/ceph.conf 2024-09-06T14:25:14.210 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:13 smithi142 ceph-mon[37372]: Updating smithi142:/etc/ceph/ceph.client.admin.keyring 2024-09-06T14:25:14.210 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:13 smithi142 ceph-mon[37372]: Updating smithi162:/etc/ceph/ceph.client.admin.keyring 2024-09-06T14:25:14.210 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:13 smithi142 ceph-mon[37372]: Updating smithi142:/var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/config/ceph.client.admin.keyring 2024-09-06T14:25:14.210 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:13 smithi142 ceph-mon[37372]: Updating smithi162:/var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/config/ceph.client.admin.keyring 2024-09-06T14:25:14.210 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:13 smithi142 ceph-mon[37372]: Deploying daemon alertmanager.a on smithi142 2024-09-06T14:25:14.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:13 smithi162 ceph-mon[33787]: from='client.14721 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-06T14:25:14.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:13 smithi162 ceph-mon[33787]: Updating smithi142:/var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/config/ceph.conf 2024-09-06T14:25:14.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:13 smithi162 ceph-mon[33787]: Updating smithi162:/var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/config/ceph.conf 2024-09-06T14:25:14.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:13 smithi162 ceph-mon[33787]: Updating smithi142:/etc/ceph/ceph.client.admin.keyring 2024-09-06T14:25:14.298 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:13 smithi162 ceph-mon[33787]: Updating smithi162:/etc/ceph/ceph.client.admin.keyring 2024-09-06T14:25:14.298 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:13 smithi162 ceph-mon[33787]: Updating smithi142:/var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/config/ceph.client.admin.keyring 2024-09-06T14:25:14.298 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:13 smithi162 ceph-mon[33787]: Updating smithi162:/var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/config/ceph.client.admin.keyring 2024-09-06T14:25:14.298 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:13 smithi162 ceph-mon[33787]: Deploying daemon alertmanager.a on smithi142 2024-09-06T14:25:14.708 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:25:14 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: ::ffff:172.21.15.162 - - [06/Sep/2024:14:25:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-06T14:25:14.996 INFO:teuthology.orchestra.run.smithi142.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2024-09-06T14:25:13.518116+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":888,"num_read_kb":745,"num_write":549,"num_write_kb":1673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":552,"ondisk_log_size":552,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":239540,"kb_used_data":6684,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":749721676,"statfs":{"total":767960285184,"available":767714996224,"internally_reserved":0,"allocated":6844416,"data_stored":3633129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003393"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":30,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.449767+0000","last_change":"2024-09-06T14:24:25.997015+0000","last_active":"2024-09-06T14:24:55.449767+0000","last_peered":"2024-09-06T14:24:55.449767+0000","last_clean":"2024-09-06T14:24:55.449767+0000","last_became_active":"2024-09-06T14:24:25.996213+0000","last_became_peered":"2024-09-06T14:24:25.996213+0000","last_unstale":"2024-09-06T14:24:55.449767+0000","last_undegraded":"2024-09-06T14:24:55.449767+0000","last_fullsized":"2024-09-06T14:24:55.449767+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_clean_scrub_stamp":"2024-09-06T14:24:24.970251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T20:14:30.434781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"60'9","reported_seq":42,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.921464+0000","last_change":"2024-09-06T14:24:28.006668+0000","last_active":"2024-09-06T14:24:55.921464+0000","last_peered":"2024-09-06T14:24:55.921464+0000","last_clean":"2024-09-06T14:24:55.921464+0000","last_became_active":"2024-09-06T14:24:28.006320+0000","last_became_peered":"2024-09-06T14:24:28.006320+0000","last_unstale":"2024-09-06T14:24:55.921464+0000","last_undegraded":"2024-09-06T14:24:55.921464+0000","last_fullsized":"2024-09-06T14:24:55.921464+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_clean_scrub_stamp":"2024-09-06T14:24:26.984416+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T19:18:48.111200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":22,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.922514+0000","last_change":"2024-09-06T14:24:30.018374+0000","last_active":"2024-09-06T14:24:55.922514+0000","last_peered":"2024-09-06T14:24:55.922514+0000","last_clean":"2024-09-06T14:24:55.922514+0000","last_became_active":"2024-09-06T14:24:30.018083+0000","last_became_peered":"2024-09-06T14:24:30.018083+0000","last_unstale":"2024-09-06T14:24:55.922514+0000","last_undegraded":"2024-09-06T14:24:55.922514+0000","last_fullsized":"2024-09-06T14:24:55.922514+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_clean_scrub_stamp":"2024-09-06T14:24:28.992633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T21:32:36.013547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":18,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.921684+0000","last_change":"2024-09-06T14:24:32.031400+0000","last_active":"2024-09-06T14:24:55.921684+0000","last_peered":"2024-09-06T14:24:55.921684+0000","last_clean":"2024-09-06T14:24:55.921684+0000","last_became_active":"2024-09-06T14:24:32.031142+0000","last_became_peered":"2024-09-06T14:24:32.031142+0000","last_unstale":"2024-09-06T14:24:55.921684+0000","last_undegraded":"2024-09-06T14:24:55.921684+0000","last_fullsized":"2024-09-06T14:24:55.921684+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_clean_scrub_stamp":"2024-09-06T14:24:31.009248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T15:39:27.650963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":18,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.449715+0000","last_change":"2024-09-06T14:24:32.038789+0000","last_active":"2024-09-06T14:24:55.449715+0000","last_peered":"2024-09-06T14:24:55.449715+0000","last_clean":"2024-09-06T14:24:55.449715+0000","last_became_active":"2024-09-06T14:24:32.038081+0000","last_became_peered":"2024-09-06T14:24:32.038081+0000","last_unstale":"2024-09-06T14:24:55.449715+0000","last_undegraded":"2024-09-06T14:24:55.449715+0000","last_fullsized":"2024-09-06T14:24:55.449715+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_clean_scrub_stamp":"2024-09-06T14:24:31.009248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T15:01:32.703233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":30,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.451483+0000","last_change":"2024-09-06T14:24:25.999476+0000","last_active":"2024-09-06T14:24:55.451483+0000","last_peered":"2024-09-06T14:24:55.451483+0000","last_clean":"2024-09-06T14:24:55.451483+0000","last_became_active":"2024-09-06T14:24:25.998488+0000","last_became_peered":"2024-09-06T14:24:25.998488+0000","last_unstale":"2024-09-06T14:24:55.451483+0000","last_undegraded":"2024-09-06T14:24:55.451483+0000","last_fullsized":"2024-09-06T14:24:55.451483+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_clean_scrub_stamp":"2024-09-06T14:24:24.970251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T23:24:57.206743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"60'15","reported_seq":51,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.450056+0000","last_change":"2024-09-06T14:24:28.014521+0000","last_active":"2024-09-06T14:24:55.450056+0000","last_peered":"2024-09-06T14:24:55.450056+0000","last_clean":"2024-09-06T14:24:55.450056+0000","last_became_active":"2024-09-06T14:24:28.014236+0000","last_became_peered":"2024-09-06T14:24:28.014236+0000","last_unstale":"2024-09-06T14:24:55.450056+0000","last_undegraded":"2024-09-06T14:24:55.450056+0000","last_fullsized":"2024-09-06T14:24:55.450056+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_clean_scrub_stamp":"2024-09-06T14:24:26.984416+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T19:33:16.100900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":22,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.921457+0000","last_change":"2024-09-06T14:24:30.017693+0000","last_active":"2024-09-06T14:24:55.921457+0000","last_peered":"2024-09-06T14:24:55.921457+0000","last_clean":"2024-09-06T14:24:55.921457+0000","last_became_active":"2024-09-06T14:24:30.017390+0000","last_became_peered":"2024-09-06T14:24:30.017390+0000","last_unstale":"2024-09-06T14:24:55.921457+0000","last_undegraded":"2024-09-06T14:24:55.921457+0000","last_fullsized":"2024-09-06T14:24:55.921457+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_clean_scrub_stamp":"2024-09-06T14:24:28.992633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T22:12:26.568897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":30,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.786297+0000","last_change":"2024-09-06T14:24:25.996508+0000","last_active":"2024-09-06T14:24:55.786297+0000","last_peered":"2024-09-06T14:24:55.786297+0000","last_clean":"2024-09-06T14:24:55.786297+0000","last_became_active":"2024-09-06T14:24:25.996217+0000","last_became_peered":"2024-09-06T14:24:25.996217+0000","last_unstale":"2024-09-06T14:24:55.786297+0000","last_undegraded":"2024-09-06T14:24:55.786297+0000","last_fullsized":"2024-09-06T14:24:55.786297+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_clean_scrub_stamp":"2024-09-06T14:24:24.970251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-08T01:15:20.055475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"60'9","reported_seq":42,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.922822+0000","last_change":"2024-09-06T14:24:28.012532+0000","last_active":"2024-09-06T14:24:55.922822+0000","last_peered":"2024-09-06T14:24:55.922822+0000","last_clean":"2024-09-06T14:24:55.922822+0000","last_became_active":"2024-09-06T14:24:28.012241+0000","last_became_peered":"2024-09-06T14:24:28.012241+0000","last_unstale":"2024-09-06T14:24:55.922822+0000","last_undegraded":"2024-09-06T14:24:55.922822+0000","last_fullsized":"2024-09-06T14:24:55.922822+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_clean_scrub_stamp":"2024-09-06T14:24:26.984416+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T15:00:50.407496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":22,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.785848+0000","last_change":"2024-09-06T14:24:30.016494+0000","last_active":"2024-09-06T14:24:55.785848+0000","last_peered":"2024-09-06T14:24:55.785848+0000","last_clean":"2024-09-06T14:24:55.785848+0000","last_became_active":"2024-09-06T14:24:30.015964+0000","last_became_peered":"2024-09-06T14:24:30.015964+0000","last_unstale":"2024-09-06T14:24:55.785848+0000","last_undegraded":"2024-09-06T14:24:55.785848+0000","last_fullsized":"2024-09-06T14:24:55.785848+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_clean_scrub_stamp":"2024-09-06T14:24:28.992633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-08T00:07:54.412112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":18,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.450243+0000","last_change":"2024-09-06T14:24:32.034268+0000","last_active":"2024-09-06T14:24:55.450243+0000","last_peered":"2024-09-06T14:24:55.450243+0000","last_clean":"2024-09-06T14:24:55.450243+0000","last_became_active":"2024-09-06T14:24:32.033955+0000","last_became_peered":"2024-09-06T14:24:32.033955+0000","last_unstale":"2024-09-06T14:24:55.450243+0000","last_undegraded":"2024-09-06T14:24:55.450243+0000","last_fullsized":"2024-09-06T14:24:55.450243+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_clean_scrub_stamp":"2024-09-06T14:24:31.009248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-08T00:37:48.342424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":30,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.786032+0000","last_change":"2024-09-06T14:24:25.997921+0000","last_active":"2024-09-06T14:24:55.786032+0000","last_peered":"2024-09-06T14:24:55.786032+0000","last_clean":"2024-09-06T14:24:55.786032+0000","last_became_active":"2024-09-06T14:24:25.997614+0000","last_became_peered":"2024-09-06T14:24:25.997614+0000","last_unstale":"2024-09-06T14:24:55.786032+0000","last_undegraded":"2024-09-06T14:24:55.786032+0000","last_fullsized":"2024-09-06T14:24:55.786032+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_clean_scrub_stamp":"2024-09-06T14:24:24.970251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-08T01:10:51.683384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"60'5","reported_seq":36,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.922035+0000","last_change":"2024-09-06T14:24:28.012417+0000","last_active":"2024-09-06T14:24:55.922035+0000","last_peered":"2024-09-06T14:24:55.922035+0000","last_clean":"2024-09-06T14:24:55.922035+0000","last_became_active":"2024-09-06T14:24:28.012080+0000","last_became_peered":"2024-09-06T14:24:28.012080+0000","last_unstale":"2024-09-06T14:24:55.922035+0000","last_undegraded":"2024-09-06T14:24:55.922035+0000","last_fullsized":"2024-09-06T14:24:55.922035+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_clean_scrub_stamp":"2024-09-06T14:24:26.984416+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T17:19:54.900883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":22,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.447179+0000","last_change":"2024-09-06T14:24:30.020516+0000","last_active":"2024-09-06T14:24:55.447179+0000","last_peered":"2024-09-06T14:24:55.447179+0000","last_clean":"2024-09-06T14:24:55.447179+0000","last_became_active":"2024-09-06T14:24:30.020075+0000","last_became_peered":"2024-09-06T14:24:30.020075+0000","last_unstale":"2024-09-06T14:24:55.447179+0000","last_undegraded":"2024-09-06T14:24:55.447179+0000","last_fullsized":"2024-09-06T14:24:55.447179+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_clean_scrub_stamp":"2024-09-06T14:24:28.992633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T21:35:05.091193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":18,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.785716+0000","last_change":"2024-09-06T14:24:32.032981+0000","last_active":"2024-09-06T14:24:55.785716+0000","last_peered":"2024-09-06T14:24:55.785716+0000","last_clean":"2024-09-06T14:24:55.785716+0000","last_became_active":"2024-09-06T14:24:32.032715+0000","last_became_peered":"2024-09-06T14:24:32.032715+0000","last_unstale":"2024-09-06T14:24:55.785716+0000","last_undegraded":"2024-09-06T14:24:55.785716+0000","last_fullsized":"2024-09-06T14:24:55.785716+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_clean_scrub_stamp":"2024-09-06T14:24:31.009248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T21:19:16.074402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":18,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.922184+0000","last_change":"2024-09-06T14:24:32.032799+0000","last_active":"2024-09-06T14:24:55.922184+0000","last_peered":"2024-09-06T14:24:55.922184+0000","last_clean":"2024-09-06T14:24:55.922184+0000","last_became_active":"2024-09-06T14:24:32.032526+0000","last_became_peered":"2024-09-06T14:24:32.032526+0000","last_unstale":"2024-09-06T14:24:55.922184+0000","last_undegraded":"2024-09-06T14:24:55.922184+0000","last_fullsized":"2024-09-06T14:24:55.922184+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_clean_scrub_stamp":"2024-09-06T14:24:31.009248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T23:59:07.742457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":30,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.450651+0000","last_change":"2024-09-06T14:24:26.002157+0000","last_active":"2024-09-06T14:24:55.450651+0000","last_peered":"2024-09-06T14:24:55.450651+0000","last_clean":"2024-09-06T14:24:55.450651+0000","last_became_active":"2024-09-06T14:24:26.001828+0000","last_became_peered":"2024-09-06T14:24:26.001828+0000","last_unstale":"2024-09-06T14:24:55.450651+0000","last_undegraded":"2024-09-06T14:24:55.450651+0000","last_fullsized":"2024-09-06T14:24:55.450651+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_clean_scrub_stamp":"2024-09-06T14:24:24.970251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T21:29:27.668185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"60'15","reported_seq":51,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.842565+0000","last_change":"2024-09-06T14:24:28.013488+0000","last_active":"2024-09-06T14:24:55.842565+0000","last_peered":"2024-09-06T14:24:55.842565+0000","last_clean":"2024-09-06T14:24:55.842565+0000","last_became_active":"2024-09-06T14:24:28.013189+0000","last_became_peered":"2024-09-06T14:24:28.013189+0000","last_unstale":"2024-09-06T14:24:55.842565+0000","last_undegraded":"2024-09-06T14:24:55.842565+0000","last_fullsized":"2024-09-06T14:24:55.842565+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_clean_scrub_stamp":"2024-09-06T14:24:26.984416+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T15:39:41.635207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":22,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.923110+0000","last_change":"2024-09-06T14:24:30.021029+0000","last_active":"2024-09-06T14:24:55.923110+0000","last_peered":"2024-09-06T14:24:55.923110+0000","last_clean":"2024-09-06T14:24:55.923110+0000","last_became_active":"2024-09-06T14:24:30.020732+0000","last_became_peered":"2024-09-06T14:24:30.020732+0000","last_unstale":"2024-09-06T14:24:55.923110+0000","last_undegraded":"2024-09-06T14:24:55.923110+0000","last_fullsized":"2024-09-06T14:24:55.923110+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_clean_scrub_stamp":"2024-09-06T14:24:28.992633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T22:25:24.936291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":18,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.451607+0000","last_change":"2024-09-06T14:24:32.036230+0000","last_active":"2024-09-06T14:24:55.451607+0000","last_peered":"2024-09-06T14:24:55.451607+0000","last_clean":"2024-09-06T14:24:55.451607+0000","last_became_active":"2024-09-06T14:24:32.035789+0000","last_became_peered":"2024-09-06T14:24:32.035789+0000","last_unstale":"2024-09-06T14:24:55.451607+0000","last_undegraded":"2024-09-06T14:24:55.451607+0000","last_fullsized":"2024-09-06T14:24:55.451607+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_clean_scrub_stamp":"2024-09-06T14:24:31.009248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T21:32:34.018012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":30,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.922898+0000","last_change":"2024-09-06T14:24:25.995904+0000","last_active":"2024-09-06T14:24:55.922898+0000","last_peered":"2024-09-06T14:24:55.922898+0000","last_clean":"2024-09-06T14:24:55.922898+0000","last_became_active":"2024-09-06T14:24:25.995503+0000","last_became_peered":"2024-09-06T14:24:25.995503+0000","last_unstale":"2024-09-06T14:24:55.922898+0000","last_undegraded":"2024-09-06T14:24:55.922898+0000","last_fullsized":"2024-09-06T14:24:55.922898+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_clean_scrub_stamp":"2024-09-06T14:24:24.970251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T21:55:52.128974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"60'12","reported_seq":49,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.451049+0000","last_change":"2024-09-06T14:24:28.017290+0000","last_active":"2024-09-06T14:24:55.451049+0000","last_peered":"2024-09-06T14:24:55.451049+0000","last_clean":"2024-09-06T14:24:55.451049+0000","last_became_active":"2024-09-06T14:24:28.016965+0000","last_became_peered":"2024-09-06T14:24:28.016965+0000","last_unstale":"2024-09-06T14:24:55.451049+0000","last_undegraded":"2024-09-06T14:24:55.451049+0000","last_fullsized":"2024-09-06T14:24:55.451049+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_clean_scrub_stamp":"2024-09-06T14:24:26.984416+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T23:50:21.062604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":22,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.922540+0000","last_change":"2024-09-06T14:24:30.018987+0000","last_active":"2024-09-06T14:24:55.922540+0000","last_peered":"2024-09-06T14:24:55.922540+0000","last_clean":"2024-09-06T14:24:55.922540+0000","last_became_active":"2024-09-06T14:24:30.018717+0000","last_became_peered":"2024-09-06T14:24:30.018717+0000","last_unstale":"2024-09-06T14:24:55.922540+0000","last_undegraded":"2024-09-06T14:24:55.922540+0000","last_fullsized":"2024-09-06T14:24:55.922540+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_clean_scrub_stamp":"2024-09-06T14:24:28.992633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-08T01:29:49.391209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"60'1","reported_seq":19,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.447576+0000","last_change":"2024-09-06T14:24:32.037082+0000","last_active":"2024-09-06T14:24:55.447576+0000","last_peered":"2024-09-06T14:24:55.447576+0000","last_clean":"2024-09-06T14:24:55.447576+0000","last_became_active":"2024-09-06T14:24:32.036782+0000","last_became_peered":"2024-09-06T14:24:32.036782+0000","last_unstale":"2024-09-06T14:24:55.447576+0000","last_undegraded":"2024-09-06T14:24:55.447576+0000","last_fullsized":"2024-09-06T14:24:55.447576+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_clean_scrub_stamp":"2024-09-06T14:24:31.009248+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-08T01:17:20.757338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"53'1","reported_seq":31,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.922669+0000","last_change":"2024-09-06T14:24:26.001573+0000","last_active":"2024-09-06T14:24:55.922669+0000","last_peered":"2024-09-06T14:24:55.922669+0000","last_clean":"2024-09-06T14:24:55.922669+0000","last_became_active":"2024-09-06T14:24:25.998395+0000","last_became_peered":"2024-09-06T14:24:25.998395+0000","last_unstale":"2024-09-06T14:24:55.922669+0000","last_undegraded":"2024-09-06T14:24:55.922669+0000","last_fullsized":"2024-09-06T14:24:55.922669+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_clean_scrub_stamp":"2024-09-06T14:24:24.970251+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T17:36:56.816449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"60'10","reported_seq":41,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.450981+0000","last_change":"2024-09-06T14:24:28.062422+0000","last_active":"2024-09-06T14:24:55.450981+0000","last_peered":"2024-09-06T14:24:55.450981+0000","last_clean":"2024-09-06T14:24:55.450981+0000","last_became_active":"2024-09-06T14:24:28.062093+0000","last_became_peered":"2024-09-06T14:24:28.062093+0000","last_unstale":"2024-09-06T14:24:55.450981+0000","last_undegraded":"2024-09-06T14:24:55.450981+0000","last_fullsized":"2024-09-06T14:24:55.450981+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_clean_scrub_stamp":"2024-09-06T14:24:26.984416+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T16:51:37.986872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"60'8","reported_seq":43,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:25:06.056629+0000","last_change":"2024-09-06T14:24:30.016621+0000","last_active":"2024-09-06T14:25:06.056629+0000","last_peered":"2024-09-06T14:25:06.056629+0000","last_clean":"2024-09-06T14:25:06.056629+0000","last_became_active":"2024-09-06T14:24:30.015760+0000","last_became_peered":"2024-09-06T14:24:30.015760+0000","last_unstale":"2024-09-06T14:25:06.056629+0000","last_undegraded":"2024-09-06T14:25:06.056629+0000","last_fullsized":"2024-09-06T14:25:06.056629+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_clean_scrub_stamp":"2024-09-06T14:24:28.992633+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T20:30:40.346775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"60'15","reported_seq":51,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.923369+0000","last_change":"2024-09-06T14:24:28.016592+0000","last_active":"2024-09-06T14:24:55.923369+0000","last_peered":"2024-09-06T14:24:55.923369+0000","last_clean":"2024-09-06T14:24:55.923369+0000","last_became_active":"2024-09-06T14:24:28.016287+0000","last_became_peered":"2024-09-06T14:24:28.016287+0000","last_unstale":"2024-09-06T14:24:55.923369+0000","last_undegraded":"2024-09-06T14:24:55.923369+0000","last_fullsized":"2024-09-06T14:24:55.923369+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_clean_scrub_stamp":"2024-09-06T14:24:26.984416+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-08T01:17:07.274863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":30,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.450852+0000","last_change":"2024-09-06T14:24:26.005250+0000","last_active":"2024-09-06T14:24:55.450852+0000","last_peered":"2024-09-06T14:24:55.450852+0000","last_clean":"2024-09-06T14:24:55.450852+0000","last_became_active":"2024-09-06T14:24:26.004932+0000","last_became_peered":"2024-09-06T14:24:26.004932+0000","last_unstale":"2024-09-06T14:24:55.450852+0000","last_undegraded":"2024-09-06T14:24:55.450852+0000","last_fullsized":"2024-09-06T14:24:55.450852+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_clean_scrub_stamp":"2024-09-06T14:24:24.970251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T15:12:03.667862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"60'8","reported_seq":40,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:25:06.056683+0000","last_change":"2024-09-06T14:24:30.011504+0000","last_active":"2024-09-06T14:25:06.056683+0000","last_peered":"2024-09-06T14:25:06.056683+0000","last_clean":"2024-09-06T14:25:06.056683+0000","last_became_active":"2024-09-06T14:24:30.011217+0000","last_became_peered":"2024-09-06T14:24:30.011217+0000","last_unstale":"2024-09-06T14:25:06.056683+0000","last_undegraded":"2024-09-06T14:25:06.056683+0000","last_fullsized":"2024-09-06T14:25:06.056683+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_clean_scrub_stamp":"2024-09-06T14:24:28.992633+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T22:14:52.780516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":18,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.785977+0000","last_change":"2024-09-06T14:24:32.034536+0000","last_active":"2024-09-06T14:24:55.785977+0000","last_peered":"2024-09-06T14:24:55.785977+0000","last_clean":"2024-09-06T14:24:55.785977+0000","last_became_active":"2024-09-06T14:24:32.034128+0000","last_became_peered":"2024-09-06T14:24:32.034128+0000","last_unstale":"2024-09-06T14:24:55.785977+0000","last_undegraded":"2024-09-06T14:24:55.785977+0000","last_fullsized":"2024-09-06T14:24:55.785977+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_clean_scrub_stamp":"2024-09-06T14:24:31.009248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T20:10:18.987767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.0","version":"60'18","reported_seq":58,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.450596+0000","last_change":"2024-09-06T14:24:28.064719+0000","last_active":"2024-09-06T14:24:55.450596+0000","last_peered":"2024-09-06T14:24:55.450596+0000","last_clean":"2024-09-06T14:24:55.450596+0000","last_became_active":"2024-09-06T14:24:28.064397+0000","last_became_peered":"2024-09-06T14:24:28.064397+0000","last_unstale":"2024-09-06T14:24:55.450596+0000","last_undegraded":"2024-09-06T14:24:55.450596+0000","last_fullsized":"2024-09-06T14:24:55.450596+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_clean_scrub_stamp":"2024-09-06T14:24:26.984416+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T21:56:32.478236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":30,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.452061+0000","last_change":"2024-09-06T14:24:26.003024+0000","last_active":"2024-09-06T14:24:55.452061+0000","last_peered":"2024-09-06T14:24:55.452061+0000","last_clean":"2024-09-06T14:24:55.452061+0000","last_became_active":"2024-09-06T14:24:26.002756+0000","last_became_peered":"2024-09-06T14:24:26.002756+0000","last_unstale":"2024-09-06T14:24:55.452061+0000","last_undegraded":"2024-09-06T14:24:55.452061+0000","last_fullsized":"2024-09-06T14:24:55.452061+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_clean_scrub_stamp":"2024-09-06T14:24:24.970251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T22:43:13.116583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":22,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.921587+0000","last_change":"2024-09-06T14:24:30.014409+0000","last_active":"2024-09-06T14:24:55.921587+0000","last_peered":"2024-09-06T14:24:55.921587+0000","last_clean":"2024-09-06T14:24:55.921587+0000","last_became_active":"2024-09-06T14:24:30.014140+0000","last_became_peered":"2024-09-06T14:24:30.014140+0000","last_unstale":"2024-09-06T14:24:55.921587+0000","last_undegraded":"2024-09-06T14:24:55.921587+0000","last_fullsized":"2024-09-06T14:24:55.921587+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_clean_scrub_stamp":"2024-09-06T14:24:28.992633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T23:53:18.617224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":18,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.922680+0000","last_change":"2024-09-06T14:24:32.036763+0000","last_active":"2024-09-06T14:24:55.922680+0000","last_peered":"2024-09-06T14:24:55.922680+0000","last_clean":"2024-09-06T14:24:55.922680+0000","last_became_active":"2024-09-06T14:24:32.036482+0000","last_became_peered":"2024-09-06T14:24:32.036482+0000","last_unstale":"2024-09-06T14:24:55.922680+0000","last_undegraded":"2024-09-06T14:24:55.922680+0000","last_fullsized":"2024-09-06T14:24:55.922680+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_clean_scrub_stamp":"2024-09-06T14:24:31.009248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T20:56:49.936018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"60'14","reported_seq":47,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.921681+0000","last_change":"2024-09-06T14:24:28.002153+0000","last_active":"2024-09-06T14:24:55.921681+0000","last_peered":"2024-09-06T14:24:55.921681+0000","last_clean":"2024-09-06T14:24:55.921681+0000","last_became_active":"2024-09-06T14:24:28.001864+0000","last_became_peered":"2024-09-06T14:24:28.001864+0000","last_unstale":"2024-09-06T14:24:55.921681+0000","last_undegraded":"2024-09-06T14:24:55.921681+0000","last_fullsized":"2024-09-06T14:24:55.921681+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_clean_scrub_stamp":"2024-09-06T14:24:26.984416+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T17:02:53.838105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"53'1","reported_seq":31,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.451342+0000","last_change":"2024-09-06T14:24:25.999108+0000","last_active":"2024-09-06T14:24:55.451342+0000","last_peered":"2024-09-06T14:24:55.451342+0000","last_clean":"2024-09-06T14:24:55.451342+0000","last_became_active":"2024-09-06T14:24:25.998791+0000","last_became_peered":"2024-09-06T14:24:25.998791+0000","last_unstale":"2024-09-06T14:24:55.451342+0000","last_undegraded":"2024-09-06T14:24:55.451342+0000","last_fullsized":"2024-09-06T14:24:55.451342+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_clean_scrub_stamp":"2024-09-06T14:24:24.970251+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T22:56:31.483325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"60'8","reported_seq":40,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:25:06.056425+0000","last_change":"2024-09-06T14:24:30.014569+0000","last_active":"2024-09-06T14:25:06.056425+0000","last_peered":"2024-09-06T14:25:06.056425+0000","last_clean":"2024-09-06T14:25:06.056425+0000","last_became_active":"2024-09-06T14:24:30.014272+0000","last_became_peered":"2024-09-06T14:24:30.014272+0000","last_unstale":"2024-09-06T14:25:06.056425+0000","last_undegraded":"2024-09-06T14:25:06.056425+0000","last_fullsized":"2024-09-06T14:25:06.056425+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_clean_scrub_stamp":"2024-09-06T14:24:28.992633+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-08T01:08:12.708501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":18,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.447320+0000","last_change":"2024-09-06T14:24:32.032068+0000","last_active":"2024-09-06T14:24:55.447320+0000","last_peered":"2024-09-06T14:24:55.447320+0000","last_clean":"2024-09-06T14:24:55.447320+0000","last_became_active":"2024-09-06T14:24:32.031785+0000","last_became_peered":"2024-09-06T14:24:32.031785+0000","last_unstale":"2024-09-06T14:24:55.447320+0000","last_undegraded":"2024-09-06T14:24:55.447320+0000","last_fullsized":"2024-09-06T14:24:55.447320+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_clean_scrub_stamp":"2024-09-06T14:24:31.009248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T18:47:27.357801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"60'10","reported_seq":41,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.922780+0000","last_change":"2024-09-06T14:24:28.005612+0000","last_active":"2024-09-06T14:24:55.922780+0000","last_peered":"2024-09-06T14:24:55.922780+0000","last_clean":"2024-09-06T14:24:55.922780+0000","last_became_active":"2024-09-06T14:24:28.005344+0000","last_became_peered":"2024-09-06T14:24:28.005344+0000","last_unstale":"2024-09-06T14:24:55.922780+0000","last_undegraded":"2024-09-06T14:24:55.922780+0000","last_fullsized":"2024-09-06T14:24:55.922780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_clean_scrub_stamp":"2024-09-06T14:24:26.984416+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T20:08:41.402288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":30,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.786295+0000","last_change":"2024-09-06T14:24:26.000038+0000","last_active":"2024-09-06T14:24:55.786295+0000","last_peered":"2024-09-06T14:24:55.786295+0000","last_clean":"2024-09-06T14:24:55.786295+0000","last_became_active":"2024-09-06T14:24:25.999753+0000","last_became_peered":"2024-09-06T14:24:25.999753+0000","last_unstale":"2024-09-06T14:24:55.786295+0000","last_undegraded":"2024-09-06T14:24:55.786295+0000","last_fullsized":"2024-09-06T14:24:55.786295+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_clean_scrub_stamp":"2024-09-06T14:24:24.970251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T16:18:07.617375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"60'8","reported_seq":43,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:25:06.056381+0000","last_change":"2024-09-06T14:24:30.014339+0000","last_active":"2024-09-06T14:25:06.056381+0000","last_peered":"2024-09-06T14:25:06.056381+0000","last_clean":"2024-09-06T14:25:06.056381+0000","last_became_active":"2024-09-06T14:24:30.014022+0000","last_became_peered":"2024-09-06T14:24:30.014022+0000","last_unstale":"2024-09-06T14:25:06.056381+0000","last_undegraded":"2024-09-06T14:25:06.056381+0000","last_fullsized":"2024-09-06T14:25:06.056381+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_clean_scrub_stamp":"2024-09-06T14:24:28.992633+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T20:24:25.037122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":18,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.450449+0000","last_change":"2024-09-06T14:24:32.033170+0000","last_active":"2024-09-06T14:24:55.450449+0000","last_peered":"2024-09-06T14:24:55.450449+0000","last_clean":"2024-09-06T14:24:55.450449+0000","last_became_active":"2024-09-06T14:24:32.032844+0000","last_became_peered":"2024-09-06T14:24:32.032844+0000","last_unstale":"2024-09-06T14:24:55.450449+0000","last_undegraded":"2024-09-06T14:24:55.450449+0000","last_fullsized":"2024-09-06T14:24:55.450449+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_clean_scrub_stamp":"2024-09-06T14:24:31.009248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T18:07:13.387892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"60'19","reported_seq":62,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.450359+0000","last_change":"2024-09-06T14:24:28.062441+0000","last_active":"2024-09-06T14:24:55.450359+0000","last_peered":"2024-09-06T14:24:55.450359+0000","last_clean":"2024-09-06T14:24:55.450359+0000","last_became_active":"2024-09-06T14:24:28.062131+0000","last_became_peered":"2024-09-06T14:24:28.062131+0000","last_unstale":"2024-09-06T14:24:55.450359+0000","last_undegraded":"2024-09-06T14:24:55.450359+0000","last_fullsized":"2024-09-06T14:24:55.450359+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_clean_scrub_stamp":"2024-09-06T14:24:26.984416+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T20:58:22.084533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":30,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.923231+0000","last_change":"2024-09-06T14:24:25.993201+0000","last_active":"2024-09-06T14:24:55.923231+0000","last_peered":"2024-09-06T14:24:55.923231+0000","last_clean":"2024-09-06T14:24:55.923231+0000","last_became_active":"2024-09-06T14:24:25.992980+0000","last_became_peered":"2024-09-06T14:24:25.992980+0000","last_unstale":"2024-09-06T14:24:55.923231+0000","last_undegraded":"2024-09-06T14:24:55.923231+0000","last_fullsized":"2024-09-06T14:24:55.923231+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_clean_scrub_stamp":"2024-09-06T14:24:24.970251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T19:24:50.416704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":22,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.921859+0000","last_change":"2024-09-06T14:24:30.024052+0000","last_active":"2024-09-06T14:24:55.921859+0000","last_peered":"2024-09-06T14:24:55.921859+0000","last_clean":"2024-09-06T14:24:55.921859+0000","last_became_active":"2024-09-06T14:24:30.023745+0000","last_became_peered":"2024-09-06T14:24:30.023745+0000","last_unstale":"2024-09-06T14:24:55.921859+0000","last_undegraded":"2024-09-06T14:24:55.921859+0000","last_fullsized":"2024-09-06T14:24:55.921859+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_clean_scrub_stamp":"2024-09-06T14:24:28.992633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T17:42:37.103554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":18,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.922312+0000","last_change":"2024-09-06T14:24:32.031320+0000","last_active":"2024-09-06T14:24:55.922312+0000","last_peered":"2024-09-06T14:24:55.922312+0000","last_clean":"2024-09-06T14:24:55.922312+0000","last_became_active":"2024-09-06T14:24:32.030500+0000","last_became_peered":"2024-09-06T14:24:32.030500+0000","last_unstale":"2024-09-06T14:24:55.922312+0000","last_undegraded":"2024-09-06T14:24:55.922312+0000","last_fullsized":"2024-09-06T14:24:55.922312+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_clean_scrub_stamp":"2024-09-06T14:24:31.009248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T20:28:34.604094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"60'28","reported_seq":84,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:25:06.056434+0000","last_change":"2024-09-06T14:24:28.015353+0000","last_active":"2024-09-06T14:25:06.056434+0000","last_peered":"2024-09-06T14:25:06.056434+0000","last_clean":"2024-09-06T14:25:06.056434+0000","last_became_active":"2024-09-06T14:24:28.015055+0000","last_became_peered":"2024-09-06T14:24:28.015055+0000","last_unstale":"2024-09-06T14:25:06.056434+0000","last_undegraded":"2024-09-06T14:25:06.056434+0000","last_fullsized":"2024-09-06T14:25:06.056434+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_clean_scrub_stamp":"2024-09-06T14:24:26.984416+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T21:28:19.010695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":30,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.922733+0000","last_change":"2024-09-06T14:24:26.001741+0000","last_active":"2024-09-06T14:24:55.922733+0000","last_peered":"2024-09-06T14:24:55.922733+0000","last_clean":"2024-09-06T14:24:55.922733+0000","last_became_active":"2024-09-06T14:24:26.001177+0000","last_became_peered":"2024-09-06T14:24:26.001177+0000","last_unstale":"2024-09-06T14:24:55.922733+0000","last_undegraded":"2024-09-06T14:24:55.922733+0000","last_fullsized":"2024-09-06T14:24:55.922733+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_clean_scrub_stamp":"2024-09-06T14:24:24.970251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T16:10:12.245622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"55'2","reported_seq":40,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.785920+0000","last_change":"2024-09-06T14:24:28.001551+0000","last_active":"2024-09-06T14:24:55.785920+0000","last_peered":"2024-09-06T14:24:55.785920+0000","last_clean":"2024-09-06T14:24:55.785920+0000","last_became_active":"2024-09-06T14:24:24.990939+0000","last_became_peered":"2024-09-06T14:24:24.990939+0000","last_unstale":"2024-09-06T14:24:55.785920+0000","last_undegraded":"2024-09-06T14:24:55.785920+0000","last_fullsized":"2024-09-06T14:24:55.785920+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:23.962291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:23.962291+0000","last_clean_scrub_stamp":"2024-09-06T14:24:23.962291+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T17:12:52.384373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011799650000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":22,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.449453+0000","last_change":"2024-09-06T14:24:30.015733+0000","last_active":"2024-09-06T14:24:55.449453+0000","last_peered":"2024-09-06T14:24:55.449453+0000","last_clean":"2024-09-06T14:24:55.449453+0000","last_became_active":"2024-09-06T14:24:30.015448+0000","last_became_peered":"2024-09-06T14:24:30.015448+0000","last_unstale":"2024-09-06T14:24:55.449453+0000","last_undegraded":"2024-09-06T14:24:55.449453+0000","last_fullsized":"2024-09-06T14:24:55.449453+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_clean_scrub_stamp":"2024-09-06T14:24:28.992633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T14:57:20.026461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":18,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.451325+0000","last_change":"2024-09-06T14:24:32.038106+0000","last_active":"2024-09-06T14:24:55.451325+0000","last_peered":"2024-09-06T14:24:55.451325+0000","last_clean":"2024-09-06T14:24:55.451325+0000","last_became_active":"2024-09-06T14:24:32.037762+0000","last_became_peered":"2024-09-06T14:24:32.037762+0000","last_unstale":"2024-09-06T14:24:55.451325+0000","last_undegraded":"2024-09-06T14:24:55.451325+0000","last_fullsized":"2024-09-06T14:24:55.451325+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_clean_scrub_stamp":"2024-09-06T14:24:31.009248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T22:25:36.424591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"60'13","reported_seq":53,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.923429+0000","last_change":"2024-09-06T14:24:28.007350+0000","last_active":"2024-09-06T14:24:55.923429+0000","last_peered":"2024-09-06T14:24:55.923429+0000","last_clean":"2024-09-06T14:24:55.923429+0000","last_became_active":"2024-09-06T14:24:28.007105+0000","last_became_peered":"2024-09-06T14:24:28.007105+0000","last_unstale":"2024-09-06T14:24:55.923429+0000","last_undegraded":"2024-09-06T14:24:55.923429+0000","last_fullsized":"2024-09-06T14:24:55.923429+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_clean_scrub_stamp":"2024-09-06T14:24:26.984416+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T19:11:11.276341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":30,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.922311+0000","last_change":"2024-09-06T14:24:25.988258+0000","last_active":"2024-09-06T14:24:55.922311+0000","last_peered":"2024-09-06T14:24:55.922311+0000","last_clean":"2024-09-06T14:24:55.922311+0000","last_became_active":"2024-09-06T14:24:25.987935+0000","last_became_peered":"2024-09-06T14:24:25.987935+0000","last_unstale":"2024-09-06T14:24:55.922311+0000","last_undegraded":"2024-09-06T14:24:55.922311+0000","last_fullsized":"2024-09-06T14:24:55.922311+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_clean_scrub_stamp":"2024-09-06T14:24:24.970251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T18:18:47.349222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"53'1","reported_seq":39,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.841772+0000","last_change":"2024-09-06T14:24:28.005058+0000","last_active":"2024-09-06T14:24:55.841772+0000","last_peered":"2024-09-06T14:24:55.841772+0000","last_clean":"2024-09-06T14:24:55.841772+0000","last_became_active":"2024-09-06T14:24:24.987965+0000","last_became_peered":"2024-09-06T14:24:24.987965+0000","last_unstale":"2024-09-06T14:24:55.841772+0000","last_undegraded":"2024-09-06T14:24:55.841772+0000","last_fullsized":"2024-09-06T14:24:55.841772+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:23.962291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:23.962291+0000","last_clean_scrub_stamp":"2024-09-06T14:24:23.962291+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T23:31:31.658900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0046516300000000003,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":22,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.842325+0000","last_change":"2024-09-06T14:24:30.019424+0000","last_active":"2024-09-06T14:24:55.842325+0000","last_peered":"2024-09-06T14:24:55.842325+0000","last_clean":"2024-09-06T14:24:55.842325+0000","last_became_active":"2024-09-06T14:24:30.019111+0000","last_became_peered":"2024-09-06T14:24:30.019111+0000","last_unstale":"2024-09-06T14:24:55.842325+0000","last_undegraded":"2024-09-06T14:24:55.842325+0000","last_fullsized":"2024-09-06T14:24:55.842325+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_clean_scrub_stamp":"2024-09-06T14:24:28.992633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T20:06:38.974950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":18,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.447811+0000","last_change":"2024-09-06T14:24:32.040295+0000","last_active":"2024-09-06T14:24:55.447811+0000","last_peered":"2024-09-06T14:24:55.447811+0000","last_clean":"2024-09-06T14:24:55.447811+0000","last_became_active":"2024-09-06T14:24:32.039953+0000","last_became_peered":"2024-09-06T14:24:32.039953+0000","last_unstale":"2024-09-06T14:24:55.447811+0000","last_undegraded":"2024-09-06T14:24:55.447811+0000","last_fullsized":"2024-09-06T14:24:55.447811+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_clean_scrub_stamp":"2024-09-06T14:24:31.009248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-08T00:45:52.313889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"60'12","reported_seq":44,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.451102+0000","last_change":"2024-09-06T14:24:28.008746+0000","last_active":"2024-09-06T14:24:55.451102+0000","last_peered":"2024-09-06T14:24:55.451102+0000","last_clean":"2024-09-06T14:24:55.451102+0000","last_became_active":"2024-09-06T14:24:28.008441+0000","last_became_peered":"2024-09-06T14:24:28.008441+0000","last_unstale":"2024-09-06T14:24:55.451102+0000","last_undegraded":"2024-09-06T14:24:55.451102+0000","last_fullsized":"2024-09-06T14:24:55.451102+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_clean_scrub_stamp":"2024-09-06T14:24:26.984416+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-08T02:14:08.659425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":30,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.448885+0000","last_change":"2024-09-06T14:24:26.005124+0000","last_active":"2024-09-06T14:24:55.448885+0000","last_peered":"2024-09-06T14:24:55.448885+0000","last_clean":"2024-09-06T14:24:55.448885+0000","last_became_active":"2024-09-06T14:24:26.004836+0000","last_became_peered":"2024-09-06T14:24:26.004836+0000","last_unstale":"2024-09-06T14:24:55.448885+0000","last_undegraded":"2024-09-06T14:24:55.448885+0000","last_fullsized":"2024-09-06T14:24:55.448885+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_clean_scrub_stamp":"2024-09-06T14:24:24.970251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T18:23:00.702043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"61'5","reported_seq":81,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:25:10.033064+0000","last_change":"2024-09-06T14:24:28.062498+0000","last_active":"2024-09-06T14:25:10.033064+0000","last_peered":"2024-09-06T14:25:10.033064+0000","last_clean":"2024-09-06T14:25:10.033064+0000","last_became_active":"2024-09-06T14:24:24.991638+0000","last_became_peered":"2024-09-06T14:24:24.991638+0000","last_unstale":"2024-09-06T14:25:10.033064+0000","last_undegraded":"2024-09-06T14:25:10.033064+0000","last_fullsized":"2024-09-06T14:25:10.033064+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:23.962291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:23.962291+0000","last_clean_scrub_stamp":"2024-09-06T14:24:23.962291+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T22:07:02.083358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061857000000000004,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":40,"num_read_kb":35,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":22,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.786384+0000","last_change":"2024-09-06T14:24:30.019269+0000","last_active":"2024-09-06T14:24:55.786384+0000","last_peered":"2024-09-06T14:24:55.786384+0000","last_clean":"2024-09-06T14:24:55.786384+0000","last_became_active":"2024-09-06T14:24:30.018880+0000","last_became_peered":"2024-09-06T14:24:30.018880+0000","last_unstale":"2024-09-06T14:24:55.786384+0000","last_undegraded":"2024-09-06T14:24:55.786384+0000","last_fullsized":"2024-09-06T14:24:55.786384+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_clean_scrub_stamp":"2024-09-06T14:24:28.992633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T19:29:09.374819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":18,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.922818+0000","last_change":"2024-09-06T14:24:32.034342+0000","last_active":"2024-09-06T14:24:55.922818+0000","last_peered":"2024-09-06T14:24:55.922818+0000","last_clean":"2024-09-06T14:24:55.922818+0000","last_became_active":"2024-09-06T14:24:32.034023+0000","last_became_peered":"2024-09-06T14:24:32.034023+0000","last_unstale":"2024-09-06T14:24:55.922818+0000","last_undegraded":"2024-09-06T14:24:55.922818+0000","last_fullsized":"2024-09-06T14:24:55.922818+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_clean_scrub_stamp":"2024-09-06T14:24:31.009248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T18:03:07.564884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"60'16","reported_seq":58,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:25:06.056689+0000","last_change":"2024-09-06T14:24:28.062894+0000","last_active":"2024-09-06T14:25:06.056689+0000","last_peered":"2024-09-06T14:25:06.056689+0000","last_clean":"2024-09-06T14:25:06.056689+0000","last_became_active":"2024-09-06T14:24:28.062494+0000","last_became_peered":"2024-09-06T14:24:28.062494+0000","last_unstale":"2024-09-06T14:25:06.056689+0000","last_undegraded":"2024-09-06T14:25:06.056689+0000","last_fullsized":"2024-09-06T14:25:06.056689+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_clean_scrub_stamp":"2024-09-06T14:24:26.984416+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T18:15:25.399135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":30,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.450048+0000","last_change":"2024-09-06T14:24:26.000957+0000","last_active":"2024-09-06T14:24:55.450048+0000","last_peered":"2024-09-06T14:24:55.450048+0000","last_clean":"2024-09-06T14:24:55.450048+0000","last_became_active":"2024-09-06T14:24:26.000658+0000","last_became_peered":"2024-09-06T14:24:26.000658+0000","last_unstale":"2024-09-06T14:24:55.450048+0000","last_undegraded":"2024-09-06T14:24:55.450048+0000","last_fullsized":"2024-09-06T14:24:55.450048+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_clean_scrub_stamp":"2024-09-06T14:24:24.970251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T18:22:03.206317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"62'79","reported_seq":67,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:57.815056+0000","last_change":"2024-09-06T14:24:19.176211+0000","last_active":"2024-09-06T14:24:57.815056+0000","last_peered":"2024-09-06T14:24:57.815056+0000","last_clean":"2024-09-06T14:24:57.815056+0000","last_became_active":"2024-09-06T14:24:19.169398+0000","last_became_peered":"2024-09-06T14:24:19.169398+0000","last_unstale":"2024-09-06T14:24:57.815056+0000","last_undegraded":"2024-09-06T14:24:57.815056+0000","last_fullsized":"2024-09-06T14:24:57.815056+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:22:11.956383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:22:11.956383+0000","last_clean_scrub_stamp":"2024-09-06T14:22:11.956383+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T16:04:11.762037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":22,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.447706+0000","last_change":"2024-09-06T14:24:30.021108+0000","last_active":"2024-09-06T14:24:55.447706+0000","last_peered":"2024-09-06T14:24:55.447706+0000","last_clean":"2024-09-06T14:24:55.447706+0000","last_became_active":"2024-09-06T14:24:30.020796+0000","last_became_peered":"2024-09-06T14:24:30.020796+0000","last_unstale":"2024-09-06T14:24:55.447706+0000","last_undegraded":"2024-09-06T14:24:55.447706+0000","last_fullsized":"2024-09-06T14:24:55.447706+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_clean_scrub_stamp":"2024-09-06T14:24:28.992633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T23:21:45.657726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":18,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.786317+0000","last_change":"2024-09-06T14:24:32.035497+0000","last_active":"2024-09-06T14:24:55.786317+0000","last_peered":"2024-09-06T14:24:55.786317+0000","last_clean":"2024-09-06T14:24:55.786317+0000","last_became_active":"2024-09-06T14:24:32.034752+0000","last_became_peered":"2024-09-06T14:24:32.034752+0000","last_unstale":"2024-09-06T14:24:55.786317+0000","last_undegraded":"2024-09-06T14:24:55.786317+0000","last_fullsized":"2024-09-06T14:24:55.786317+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_clean_scrub_stamp":"2024-09-06T14:24:31.009248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T23:19:54.020180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"60'11","reported_seq":45,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.922959+0000","last_change":"2024-09-06T14:24:28.004795+0000","last_active":"2024-09-06T14:24:55.922959+0000","last_peered":"2024-09-06T14:24:55.922959+0000","last_clean":"2024-09-06T14:24:55.922959+0000","last_became_active":"2024-09-06T14:24:28.004411+0000","last_became_peered":"2024-09-06T14:24:28.004411+0000","last_unstale":"2024-09-06T14:24:55.922959+0000","last_undegraded":"2024-09-06T14:24:55.922959+0000","last_fullsized":"2024-09-06T14:24:55.922959+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_clean_scrub_stamp":"2024-09-06T14:24:26.984416+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T17:15:55.813063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":30,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.922131+0000","last_change":"2024-09-06T14:24:26.000813+0000","last_active":"2024-09-06T14:24:55.922131+0000","last_peered":"2024-09-06T14:24:55.922131+0000","last_clean":"2024-09-06T14:24:55.922131+0000","last_became_active":"2024-09-06T14:24:26.000247+0000","last_became_peered":"2024-09-06T14:24:26.000247+0000","last_unstale":"2024-09-06T14:24:55.922131+0000","last_undegraded":"2024-09-06T14:24:55.922131+0000","last_fullsized":"2024-09-06T14:24:55.922131+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_clean_scrub_stamp":"2024-09-06T14:24:24.970251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T19:34:42.146507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":22,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.786223+0000","last_change":"2024-09-06T14:24:30.013658+0000","last_active":"2024-09-06T14:24:55.786223+0000","last_peered":"2024-09-06T14:24:55.786223+0000","last_clean":"2024-09-06T14:24:55.786223+0000","last_became_active":"2024-09-06T14:24:30.013334+0000","last_became_peered":"2024-09-06T14:24:30.013334+0000","last_unstale":"2024-09-06T14:24:55.786223+0000","last_undegraded":"2024-09-06T14:24:55.786223+0000","last_fullsized":"2024-09-06T14:24:55.786223+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_clean_scrub_stamp":"2024-09-06T14:24:28.992633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T21:28:56.142218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":18,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.450135+0000","last_change":"2024-09-06T14:24:32.035785+0000","last_active":"2024-09-06T14:24:55.450135+0000","last_peered":"2024-09-06T14:24:55.450135+0000","last_clean":"2024-09-06T14:24:55.450135+0000","last_became_active":"2024-09-06T14:24:32.035484+0000","last_became_peered":"2024-09-06T14:24:32.035484+0000","last_unstale":"2024-09-06T14:24:55.450135+0000","last_undegraded":"2024-09-06T14:24:55.450135+0000","last_fullsized":"2024-09-06T14:24:55.450135+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_clean_scrub_stamp":"2024-09-06T14:24:31.009248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-08T00:29:42.639305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"60'17","reported_seq":54,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.922370+0000","last_change":"2024-09-06T14:24:28.008597+0000","last_active":"2024-09-06T14:24:55.922370+0000","last_peered":"2024-09-06T14:24:55.922370+0000","last_clean":"2024-09-06T14:24:55.922370+0000","last_became_active":"2024-09-06T14:24:28.008244+0000","last_became_peered":"2024-09-06T14:24:28.008244+0000","last_unstale":"2024-09-06T14:24:55.922370+0000","last_undegraded":"2024-09-06T14:24:55.922370+0000","last_fullsized":"2024-09-06T14:24:55.922370+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_clean_scrub_stamp":"2024-09-06T14:24:26.984416+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-08T00:38:37.360978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"53'1","reported_seq":36,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.922301+0000","last_change":"2024-09-06T14:24:25.994258+0000","last_active":"2024-09-06T14:24:55.922301+0000","last_peered":"2024-09-06T14:24:55.922301+0000","last_clean":"2024-09-06T14:24:55.922301+0000","last_became_active":"2024-09-06T14:24:25.993164+0000","last_became_peered":"2024-09-06T14:24:25.993164+0000","last_unstale":"2024-09-06T14:24:55.922301+0000","last_undegraded":"2024-09-06T14:24:55.922301+0000","last_fullsized":"2024-09-06T14:24:55.922301+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_clean_scrub_stamp":"2024-09-06T14:24:24.970251+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T23:16:28.120890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":22,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.922909+0000","last_change":"2024-09-06T14:24:30.020695+0000","last_active":"2024-09-06T14:24:55.922909+0000","last_peered":"2024-09-06T14:24:55.922909+0000","last_clean":"2024-09-06T14:24:55.922909+0000","last_became_active":"2024-09-06T14:24:30.020195+0000","last_became_peered":"2024-09-06T14:24:30.020195+0000","last_unstale":"2024-09-06T14:24:55.922909+0000","last_undegraded":"2024-09-06T14:24:55.922909+0000","last_fullsized":"2024-09-06T14:24:55.922909+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_clean_scrub_stamp":"2024-09-06T14:24:28.992633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-08T01:13:58.867798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":18,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.842219+0000","last_change":"2024-09-06T14:24:32.034092+0000","last_active":"2024-09-06T14:24:55.842219+0000","last_peered":"2024-09-06T14:24:55.842219+0000","last_clean":"2024-09-06T14:24:55.842219+0000","last_became_active":"2024-09-06T14:24:32.033678+0000","last_became_peered":"2024-09-06T14:24:32.033678+0000","last_unstale":"2024-09-06T14:24:55.842219+0000","last_undegraded":"2024-09-06T14:24:55.842219+0000","last_fullsized":"2024-09-06T14:24:55.842219+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_clean_scrub_stamp":"2024-09-06T14:24:31.009248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T23:11:45.663142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"60'10","reported_seq":41,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.922634+0000","last_change":"2024-09-06T14:24:28.013847+0000","last_active":"2024-09-06T14:24:55.922634+0000","last_peered":"2024-09-06T14:24:55.922634+0000","last_clean":"2024-09-06T14:24:55.922634+0000","last_became_active":"2024-09-06T14:24:28.013287+0000","last_became_peered":"2024-09-06T14:24:28.013287+0000","last_unstale":"2024-09-06T14:24:55.922634+0000","last_undegraded":"2024-09-06T14:24:55.922634+0000","last_fullsized":"2024-09-06T14:24:55.922634+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_clean_scrub_stamp":"2024-09-06T14:24:26.984416+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T15:18:46.754706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":30,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.450173+0000","last_change":"2024-09-06T14:24:26.003545+0000","last_active":"2024-09-06T14:24:55.450173+0000","last_peered":"2024-09-06T14:24:55.450173+0000","last_clean":"2024-09-06T14:24:55.450173+0000","last_became_active":"2024-09-06T14:24:26.003278+0000","last_became_peered":"2024-09-06T14:24:26.003278+0000","last_unstale":"2024-09-06T14:24:55.450173+0000","last_undegraded":"2024-09-06T14:24:55.450173+0000","last_fullsized":"2024-09-06T14:24:55.450173+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_clean_scrub_stamp":"2024-09-06T14:24:24.970251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-08T01:01:43.104524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"60'8","reported_seq":40,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:25:06.056414+0000","last_change":"2024-09-06T14:24:30.021484+0000","last_active":"2024-09-06T14:25:06.056414+0000","last_peered":"2024-09-06T14:25:06.056414+0000","last_clean":"2024-09-06T14:25:06.056414+0000","last_became_active":"2024-09-06T14:24:30.021129+0000","last_became_peered":"2024-09-06T14:24:30.021129+0000","last_unstale":"2024-09-06T14:25:06.056414+0000","last_undegraded":"2024-09-06T14:25:06.056414+0000","last_fullsized":"2024-09-06T14:25:06.056414+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_clean_scrub_stamp":"2024-09-06T14:24:28.992633+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T23:03:00.443813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"60'1","reported_seq":19,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.922058+0000","last_change":"2024-09-06T14:24:32.030737+0000","last_active":"2024-09-06T14:24:55.922058+0000","last_peered":"2024-09-06T14:24:55.922058+0000","last_clean":"2024-09-06T14:24:55.922058+0000","last_became_active":"2024-09-06T14:24:32.030434+0000","last_became_peered":"2024-09-06T14:24:32.030434+0000","last_unstale":"2024-09-06T14:24:55.922058+0000","last_undegraded":"2024-09-06T14:24:55.922058+0000","last_fullsized":"2024-09-06T14:24:55.922058+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_clean_scrub_stamp":"2024-09-06T14:24:31.009248+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T16:21:06.363040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"60'9","reported_seq":42,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.450234+0000","last_change":"2024-09-06T14:24:28.005629+0000","last_active":"2024-09-06T14:24:55.450234+0000","last_peered":"2024-09-06T14:24:55.450234+0000","last_clean":"2024-09-06T14:24:55.450234+0000","last_became_active":"2024-09-06T14:24:28.005238+0000","last_became_peered":"2024-09-06T14:24:28.005238+0000","last_unstale":"2024-09-06T14:24:55.450234+0000","last_undegraded":"2024-09-06T14:24:55.450234+0000","last_fullsized":"2024-09-06T14:24:55.450234+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_clean_scrub_stamp":"2024-09-06T14:24:26.984416+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T21:10:02.684938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":30,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.786105+0000","last_change":"2024-09-06T14:24:25.999292+0000","last_active":"2024-09-06T14:24:55.786105+0000","last_peered":"2024-09-06T14:24:55.786105+0000","last_clean":"2024-09-06T14:24:55.786105+0000","last_became_active":"2024-09-06T14:24:25.999018+0000","last_became_peered":"2024-09-06T14:24:25.999018+0000","last_unstale":"2024-09-06T14:24:55.786105+0000","last_undegraded":"2024-09-06T14:24:55.786105+0000","last_fullsized":"2024-09-06T14:24:55.786105+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_clean_scrub_stamp":"2024-09-06T14:24:24.970251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T20:23:44.349519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":22,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.841713+0000","last_change":"2024-09-06T14:24:30.020394+0000","last_active":"2024-09-06T14:24:55.841713+0000","last_peered":"2024-09-06T14:24:55.841713+0000","last_clean":"2024-09-06T14:24:55.841713+0000","last_became_active":"2024-09-06T14:24:30.020089+0000","last_became_peered":"2024-09-06T14:24:30.020089+0000","last_unstale":"2024-09-06T14:24:55.841713+0000","last_undegraded":"2024-09-06T14:24:55.841713+0000","last_fullsized":"2024-09-06T14:24:55.841713+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_clean_scrub_stamp":"2024-09-06T14:24:28.992633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-08T00:30:08.037871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":18,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.448682+0000","last_change":"2024-09-06T14:24:32.032306+0000","last_active":"2024-09-06T14:24:55.448682+0000","last_peered":"2024-09-06T14:24:55.448682+0000","last_clean":"2024-09-06T14:24:55.448682+0000","last_became_active":"2024-09-06T14:24:32.032004+0000","last_became_peered":"2024-09-06T14:24:32.032004+0000","last_unstale":"2024-09-06T14:24:55.448682+0000","last_undegraded":"2024-09-06T14:24:55.448682+0000","last_fullsized":"2024-09-06T14:24:55.448682+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_clean_scrub_stamp":"2024-09-06T14:24:31.009248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-08T01:09:45.306197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"60'19","reported_seq":57,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.921962+0000","last_change":"2024-09-06T14:24:28.064397+0000","last_active":"2024-09-06T14:24:55.921962+0000","last_peered":"2024-09-06T14:24:55.921962+0000","last_clean":"2024-09-06T14:24:55.921962+0000","last_became_active":"2024-09-06T14:24:28.064064+0000","last_became_peered":"2024-09-06T14:24:28.064064+0000","last_unstale":"2024-09-06T14:24:55.921962+0000","last_undegraded":"2024-09-06T14:24:55.921962+0000","last_fullsized":"2024-09-06T14:24:55.921962+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_clean_scrub_stamp":"2024-09-06T14:24:26.984416+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T16:29:03.138477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":30,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.448222+0000","last_change":"2024-09-06T14:24:25.996040+0000","last_active":"2024-09-06T14:24:55.448222+0000","last_peered":"2024-09-06T14:24:55.448222+0000","last_clean":"2024-09-06T14:24:55.448222+0000","last_became_active":"2024-09-06T14:24:25.995769+0000","last_became_peered":"2024-09-06T14:24:25.995769+0000","last_unstale":"2024-09-06T14:24:55.448222+0000","last_undegraded":"2024-09-06T14:24:55.448222+0000","last_fullsized":"2024-09-06T14:24:55.448222+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_clean_scrub_stamp":"2024-09-06T14:24:24.970251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T20:43:32.771519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":22,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.841955+0000","last_change":"2024-09-06T14:24:30.024892+0000","last_active":"2024-09-06T14:24:55.841955+0000","last_peered":"2024-09-06T14:24:55.841955+0000","last_clean":"2024-09-06T14:24:55.841955+0000","last_became_active":"2024-09-06T14:24:30.024549+0000","last_became_peered":"2024-09-06T14:24:30.024549+0000","last_unstale":"2024-09-06T14:24:55.841955+0000","last_undegraded":"2024-09-06T14:24:55.841955+0000","last_fullsized":"2024-09-06T14:24:55.841955+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_clean_scrub_stamp":"2024-09-06T14:24:28.992633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T23:11:05.742408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":18,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.447142+0000","last_change":"2024-09-06T14:24:32.039789+0000","last_active":"2024-09-06T14:24:55.447142+0000","last_peered":"2024-09-06T14:24:55.447142+0000","last_clean":"2024-09-06T14:24:55.447142+0000","last_became_active":"2024-09-06T14:24:32.039582+0000","last_became_peered":"2024-09-06T14:24:32.039582+0000","last_unstale":"2024-09-06T14:24:55.447142+0000","last_undegraded":"2024-09-06T14:24:55.447142+0000","last_fullsized":"2024-09-06T14:24:55.447142+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_clean_scrub_stamp":"2024-09-06T14:24:31.009248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T15:15:24.485063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"60'12","reported_seq":49,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.921931+0000","last_change":"2024-09-06T14:24:28.013600+0000","last_active":"2024-09-06T14:24:55.921931+0000","last_peered":"2024-09-06T14:24:55.921931+0000","last_clean":"2024-09-06T14:24:55.921931+0000","last_became_active":"2024-09-06T14:24:28.013228+0000","last_became_peered":"2024-09-06T14:24:28.013228+0000","last_unstale":"2024-09-06T14:24:55.921931+0000","last_undegraded":"2024-09-06T14:24:55.921931+0000","last_fullsized":"2024-09-06T14:24:55.921931+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_clean_scrub_stamp":"2024-09-06T14:24:26.984416+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T16:55:36.416197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":30,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.448774+0000","last_change":"2024-09-06T14:24:25.999147+0000","last_active":"2024-09-06T14:24:55.448774+0000","last_peered":"2024-09-06T14:24:55.448774+0000","last_clean":"2024-09-06T14:24:55.448774+0000","last_became_active":"2024-09-06T14:24:25.998793+0000","last_became_peered":"2024-09-06T14:24:25.998793+0000","last_unstale":"2024-09-06T14:24:55.448774+0000","last_undegraded":"2024-09-06T14:24:55.448774+0000","last_fullsized":"2024-09-06T14:24:55.448774+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_clean_scrub_stamp":"2024-09-06T14:24:24.970251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T15:58:20.551192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":22,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.842074+0000","last_change":"2024-09-06T14:24:30.024147+0000","last_active":"2024-09-06T14:24:55.842074+0000","last_peered":"2024-09-06T14:24:55.842074+0000","last_clean":"2024-09-06T14:24:55.842074+0000","last_became_active":"2024-09-06T14:24:30.023785+0000","last_became_peered":"2024-09-06T14:24:30.023785+0000","last_unstale":"2024-09-06T14:24:55.842074+0000","last_undegraded":"2024-09-06T14:24:55.842074+0000","last_fullsized":"2024-09-06T14:24:55.842074+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_clean_scrub_stamp":"2024-09-06T14:24:28.992633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T17:19:53.871358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":18,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.449826+0000","last_change":"2024-09-06T14:24:32.039389+0000","last_active":"2024-09-06T14:24:55.449826+0000","last_peered":"2024-09-06T14:24:55.449826+0000","last_clean":"2024-09-06T14:24:55.449826+0000","last_became_active":"2024-09-06T14:24:32.038270+0000","last_became_peered":"2024-09-06T14:24:32.038270+0000","last_unstale":"2024-09-06T14:24:55.449826+0000","last_undegraded":"2024-09-06T14:24:55.449826+0000","last_fullsized":"2024-09-06T14:24:55.449826+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_clean_scrub_stamp":"2024-09-06T14:24:31.009248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T15:52:23.975701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"60'15","reported_seq":51,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.785619+0000","last_change":"2024-09-06T14:24:28.062462+0000","last_active":"2024-09-06T14:24:55.785619+0000","last_peered":"2024-09-06T14:24:55.785619+0000","last_clean":"2024-09-06T14:24:55.785619+0000","last_became_active":"2024-09-06T14:24:28.062144+0000","last_became_peered":"2024-09-06T14:24:28.062144+0000","last_unstale":"2024-09-06T14:24:55.785619+0000","last_undegraded":"2024-09-06T14:24:55.785619+0000","last_fullsized":"2024-09-06T14:24:55.785619+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_clean_scrub_stamp":"2024-09-06T14:24:26.984416+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-08T02:12:19.975625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"53'2","reported_seq":42,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.448443+0000","last_change":"2024-09-06T14:24:26.001845+0000","last_active":"2024-09-06T14:24:55.448443+0000","last_peered":"2024-09-06T14:24:55.448443+0000","last_clean":"2024-09-06T14:24:55.448443+0000","last_became_active":"2024-09-06T14:24:26.001579+0000","last_became_peered":"2024-09-06T14:24:26.001579+0000","last_unstale":"2024-09-06T14:24:55.448443+0000","last_undegraded":"2024-09-06T14:24:55.448443+0000","last_fullsized":"2024-09-06T14:24:55.448443+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_clean_scrub_stamp":"2024-09-06T14:24:24.970251+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-08T01:53:37.000913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"60'8","reported_seq":40,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:25:06.056621+0000","last_change":"2024-09-06T14:24:30.019173+0000","last_active":"2024-09-06T14:25:06.056621+0000","last_peered":"2024-09-06T14:25:06.056621+0000","last_clean":"2024-09-06T14:25:06.056621+0000","last_became_active":"2024-09-06T14:24:30.018792+0000","last_became_peered":"2024-09-06T14:24:30.018792+0000","last_unstale":"2024-09-06T14:25:06.056621+0000","last_undegraded":"2024-09-06T14:25:06.056621+0000","last_fullsized":"2024-09-06T14:25:06.056621+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_clean_scrub_stamp":"2024-09-06T14:24:28.992633+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T19:46:37.295207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":18,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.785728+0000","last_change":"2024-09-06T14:24:32.032978+0000","last_active":"2024-09-06T14:24:55.785728+0000","last_peered":"2024-09-06T14:24:55.785728+0000","last_clean":"2024-09-06T14:24:55.785728+0000","last_became_active":"2024-09-06T14:24:32.032701+0000","last_became_peered":"2024-09-06T14:24:32.032701+0000","last_unstale":"2024-09-06T14:24:55.785728+0000","last_undegraded":"2024-09-06T14:24:55.785728+0000","last_fullsized":"2024-09-06T14:24:55.785728+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_clean_scrub_stamp":"2024-09-06T14:24:31.009248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T16:38:24.537554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":30,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.921719+0000","last_change":"2024-09-06T14:24:25.994783+0000","last_active":"2024-09-06T14:24:55.921719+0000","last_peered":"2024-09-06T14:24:55.921719+0000","last_clean":"2024-09-06T14:24:55.921719+0000","last_became_active":"2024-09-06T14:24:25.994514+0000","last_became_peered":"2024-09-06T14:24:25.994514+0000","last_unstale":"2024-09-06T14:24:55.921719+0000","last_undegraded":"2024-09-06T14:24:55.921719+0000","last_fullsized":"2024-09-06T14:24:55.921719+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_clean_scrub_stamp":"2024-09-06T14:24:24.970251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T17:55:21.826188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"60'6","reported_seq":35,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.451831+0000","last_change":"2024-09-06T14:24:28.065250+0000","last_active":"2024-09-06T14:24:55.451831+0000","last_peered":"2024-09-06T14:24:55.451831+0000","last_clean":"2024-09-06T14:24:55.451831+0000","last_became_active":"2024-09-06T14:24:28.064819+0000","last_became_peered":"2024-09-06T14:24:28.064819+0000","last_unstale":"2024-09-06T14:24:55.451831+0000","last_undegraded":"2024-09-06T14:24:55.451831+0000","last_fullsized":"2024-09-06T14:24:55.451831+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_clean_scrub_stamp":"2024-09-06T14:24:26.984416+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-08T01:01:44.441868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":22,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.786130+0000","last_change":"2024-09-06T14:24:30.018751+0000","last_active":"2024-09-06T14:24:55.786130+0000","last_peered":"2024-09-06T14:24:55.786130+0000","last_clean":"2024-09-06T14:24:55.786130+0000","last_became_active":"2024-09-06T14:24:30.018294+0000","last_became_peered":"2024-09-06T14:24:30.018294+0000","last_unstale":"2024-09-06T14:24:55.786130+0000","last_undegraded":"2024-09-06T14:24:55.786130+0000","last_fullsized":"2024-09-06T14:24:55.786130+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_clean_scrub_stamp":"2024-09-06T14:24:28.992633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T19:52:16.089427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":18,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.447624+0000","last_change":"2024-09-06T14:24:32.031674+0000","last_active":"2024-09-06T14:24:55.447624+0000","last_peered":"2024-09-06T14:24:55.447624+0000","last_clean":"2024-09-06T14:24:55.447624+0000","last_became_active":"2024-09-06T14:24:32.030598+0000","last_became_peered":"2024-09-06T14:24:32.030598+0000","last_unstale":"2024-09-06T14:24:55.447624+0000","last_undegraded":"2024-09-06T14:24:55.447624+0000","last_fullsized":"2024-09-06T14:24:55.447624+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_clean_scrub_stamp":"2024-09-06T14:24:31.009248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T23:16:38.282134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"60'9","reported_seq":42,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.451230+0000","last_change":"2024-09-06T14:24:28.064252+0000","last_active":"2024-09-06T14:24:55.451230+0000","last_peered":"2024-09-06T14:24:55.451230+0000","last_clean":"2024-09-06T14:24:55.451230+0000","last_became_active":"2024-09-06T14:24:28.063935+0000","last_became_peered":"2024-09-06T14:24:28.063935+0000","last_unstale":"2024-09-06T14:24:55.451230+0000","last_undegraded":"2024-09-06T14:24:55.451230+0000","last_fullsized":"2024-09-06T14:24:55.451230+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_clean_scrub_stamp":"2024-09-06T14:24:26.984416+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T18:19:35.669936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":30,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.447253+0000","last_change":"2024-09-06T14:24:25.997064+0000","last_active":"2024-09-06T14:24:55.447253+0000","last_peered":"2024-09-06T14:24:55.447253+0000","last_clean":"2024-09-06T14:24:55.447253+0000","last_became_active":"2024-09-06T14:24:25.996789+0000","last_became_peered":"2024-09-06T14:24:25.996789+0000","last_unstale":"2024-09-06T14:24:55.447253+0000","last_undegraded":"2024-09-06T14:24:55.447253+0000","last_fullsized":"2024-09-06T14:24:55.447253+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_clean_scrub_stamp":"2024-09-06T14:24:24.970251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T14:27:54.007057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":22,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.451692+0000","last_change":"2024-09-06T14:24:30.017003+0000","last_active":"2024-09-06T14:24:55.451692+0000","last_peered":"2024-09-06T14:24:55.451692+0000","last_clean":"2024-09-06T14:24:55.451692+0000","last_became_active":"2024-09-06T14:24:30.016696+0000","last_became_peered":"2024-09-06T14:24:30.016696+0000","last_unstale":"2024-09-06T14:24:55.451692+0000","last_undegraded":"2024-09-06T14:24:55.451692+0000","last_fullsized":"2024-09-06T14:24:55.451692+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_clean_scrub_stamp":"2024-09-06T14:24:28.992633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-08T01:02:32.539435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":18,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.842289+0000","last_change":"2024-09-06T14:24:32.033331+0000","last_active":"2024-09-06T14:24:55.842289+0000","last_peered":"2024-09-06T14:24:55.842289+0000","last_clean":"2024-09-06T14:24:55.842289+0000","last_became_active":"2024-09-06T14:24:32.033040+0000","last_became_peered":"2024-09-06T14:24:32.033040+0000","last_unstale":"2024-09-06T14:24:55.842289+0000","last_undegraded":"2024-09-06T14:24:55.842289+0000","last_fullsized":"2024-09-06T14:24:55.842289+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_clean_scrub_stamp":"2024-09-06T14:24:31.009248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T18:40:02.034547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"60'9","reported_seq":42,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.786213+0000","last_change":"2024-09-06T14:24:28.064398+0000","last_active":"2024-09-06T14:24:55.786213+0000","last_peered":"2024-09-06T14:24:55.786213+0000","last_clean":"2024-09-06T14:24:55.786213+0000","last_became_active":"2024-09-06T14:24:28.064115+0000","last_became_peered":"2024-09-06T14:24:28.064115+0000","last_unstale":"2024-09-06T14:24:55.786213+0000","last_undegraded":"2024-09-06T14:24:55.786213+0000","last_fullsized":"2024-09-06T14:24:55.786213+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_clean_scrub_stamp":"2024-09-06T14:24:26.984416+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-08T01:33:05.303141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":30,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.449067+0000","last_change":"2024-09-06T14:24:26.001890+0000","last_active":"2024-09-06T14:24:55.449067+0000","last_peered":"2024-09-06T14:24:55.449067+0000","last_clean":"2024-09-06T14:24:55.449067+0000","last_became_active":"2024-09-06T14:24:26.001120+0000","last_became_peered":"2024-09-06T14:24:26.001120+0000","last_unstale":"2024-09-06T14:24:55.449067+0000","last_undegraded":"2024-09-06T14:24:55.449067+0000","last_fullsized":"2024-09-06T14:24:55.449067+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_clean_scrub_stamp":"2024-09-06T14:24:24.970251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T18:31:07.879304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"60'8","reported_seq":40,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:25:06.056622+0000","last_change":"2024-09-06T14:24:30.020387+0000","last_active":"2024-09-06T14:25:06.056622+0000","last_peered":"2024-09-06T14:25:06.056622+0000","last_clean":"2024-09-06T14:25:06.056622+0000","last_became_active":"2024-09-06T14:24:30.020097+0000","last_became_peered":"2024-09-06T14:24:30.020097+0000","last_unstale":"2024-09-06T14:25:06.056622+0000","last_undegraded":"2024-09-06T14:25:06.056622+0000","last_fullsized":"2024-09-06T14:25:06.056622+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_clean_scrub_stamp":"2024-09-06T14:24:28.992633+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-08T01:50:20.984268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":18,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.922349+0000","last_change":"2024-09-06T14:24:32.034910+0000","last_active":"2024-09-06T14:24:55.922349+0000","last_peered":"2024-09-06T14:24:55.922349+0000","last_clean":"2024-09-06T14:24:55.922349+0000","last_became_active":"2024-09-06T14:24:32.034628+0000","last_became_peered":"2024-09-06T14:24:32.034628+0000","last_unstale":"2024-09-06T14:24:55.922349+0000","last_undegraded":"2024-09-06T14:24:55.922349+0000","last_fullsized":"2024-09-06T14:24:55.922349+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_clean_scrub_stamp":"2024-09-06T14:24:31.009248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-08T01:59:51.503411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"60'10","reported_seq":41,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.450411+0000","last_change":"2024-09-06T14:24:28.062417+0000","last_active":"2024-09-06T14:24:55.450411+0000","last_peered":"2024-09-06T14:24:55.450411+0000","last_clean":"2024-09-06T14:24:55.450411+0000","last_became_active":"2024-09-06T14:24:28.062108+0000","last_became_peered":"2024-09-06T14:24:28.062108+0000","last_unstale":"2024-09-06T14:24:55.450411+0000","last_undegraded":"2024-09-06T14:24:55.450411+0000","last_fullsized":"2024-09-06T14:24:55.450411+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_clean_scrub_stamp":"2024-09-06T14:24:26.984416+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T21:52:05.503683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":30,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.447607+0000","last_change":"2024-09-06T14:24:25.997813+0000","last_active":"2024-09-06T14:24:55.447607+0000","last_peered":"2024-09-06T14:24:55.447607+0000","last_clean":"2024-09-06T14:24:55.447607+0000","last_became_active":"2024-09-06T14:24:25.997546+0000","last_became_peered":"2024-09-06T14:24:25.997546+0000","last_unstale":"2024-09-06T14:24:55.447607+0000","last_undegraded":"2024-09-06T14:24:55.447607+0000","last_fullsized":"2024-09-06T14:24:55.447607+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_clean_scrub_stamp":"2024-09-06T14:24:24.970251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T20:13:00.222134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"60'8","reported_seq":40,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:25:06.056657+0000","last_change":"2024-09-06T14:24:30.018657+0000","last_active":"2024-09-06T14:25:06.056657+0000","last_peered":"2024-09-06T14:25:06.056657+0000","last_clean":"2024-09-06T14:25:06.056657+0000","last_became_active":"2024-09-06T14:24:30.018356+0000","last_became_peered":"2024-09-06T14:24:30.018356+0000","last_unstale":"2024-09-06T14:25:06.056657+0000","last_undegraded":"2024-09-06T14:25:06.056657+0000","last_fullsized":"2024-09-06T14:25:06.056657+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_clean_scrub_stamp":"2024-09-06T14:24:28.992633+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-08T00:47:58.743872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":18,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.450745+0000","last_change":"2024-09-06T14:24:32.032991+0000","last_active":"2024-09-06T14:24:55.450745+0000","last_peered":"2024-09-06T14:24:55.450745+0000","last_clean":"2024-09-06T14:24:55.450745+0000","last_became_active":"2024-09-06T14:24:32.032691+0000","last_became_peered":"2024-09-06T14:24:32.032691+0000","last_unstale":"2024-09-06T14:24:55.450745+0000","last_undegraded":"2024-09-06T14:24:55.450745+0000","last_fullsized":"2024-09-06T14:24:55.450745+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_clean_scrub_stamp":"2024-09-06T14:24:31.009248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T22:17:45.055713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"60'11","reported_seq":45,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.922163+0000","last_change":"2024-09-06T14:24:28.004573+0000","last_active":"2024-09-06T14:24:55.922163+0000","last_peered":"2024-09-06T14:24:55.922163+0000","last_clean":"2024-09-06T14:24:55.922163+0000","last_became_active":"2024-09-06T14:24:28.004104+0000","last_became_peered":"2024-09-06T14:24:28.004104+0000","last_unstale":"2024-09-06T14:24:55.922163+0000","last_undegraded":"2024-09-06T14:24:55.922163+0000","last_fullsized":"2024-09-06T14:24:55.922163+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_clean_scrub_stamp":"2024-09-06T14:24:26.984416+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T15:30:21.462868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":30,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.922850+0000","last_change":"2024-09-06T14:24:26.001351+0000","last_active":"2024-09-06T14:24:55.922850+0000","last_peered":"2024-09-06T14:24:55.922850+0000","last_clean":"2024-09-06T14:24:55.922850+0000","last_became_active":"2024-09-06T14:24:26.000152+0000","last_became_peered":"2024-09-06T14:24:26.000152+0000","last_unstale":"2024-09-06T14:24:55.922850+0000","last_undegraded":"2024-09-06T14:24:55.922850+0000","last_fullsized":"2024-09-06T14:24:55.922850+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_clean_scrub_stamp":"2024-09-06T14:24:24.970251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-08T00:27:15.911914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":22,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.922627+0000","last_change":"2024-09-06T14:24:30.019277+0000","last_active":"2024-09-06T14:24:55.922627+0000","last_peered":"2024-09-06T14:24:55.922627+0000","last_clean":"2024-09-06T14:24:55.922627+0000","last_became_active":"2024-09-06T14:24:30.018902+0000","last_became_peered":"2024-09-06T14:24:30.018902+0000","last_unstale":"2024-09-06T14:24:55.922627+0000","last_undegraded":"2024-09-06T14:24:55.922627+0000","last_fullsized":"2024-09-06T14:24:55.922627+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_clean_scrub_stamp":"2024-09-06T14:24:28.992633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T19:36:50.484871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":18,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.449549+0000","last_change":"2024-09-06T14:24:32.037128+0000","last_active":"2024-09-06T14:24:55.449549+0000","last_peered":"2024-09-06T14:24:55.449549+0000","last_clean":"2024-09-06T14:24:55.449549+0000","last_became_active":"2024-09-06T14:24:32.036224+0000","last_became_peered":"2024-09-06T14:24:32.036224+0000","last_unstale":"2024-09-06T14:24:55.449549+0000","last_undegraded":"2024-09-06T14:24:55.449549+0000","last_fullsized":"2024-09-06T14:24:55.449549+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_clean_scrub_stamp":"2024-09-06T14:24:31.009248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-08T00:02:25.670231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"60'9","reported_seq":42,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.922923+0000","last_change":"2024-09-06T14:24:28.009721+0000","last_active":"2024-09-06T14:24:55.922923+0000","last_peered":"2024-09-06T14:24:55.922923+0000","last_clean":"2024-09-06T14:24:55.922923+0000","last_became_active":"2024-09-06T14:24:28.009420+0000","last_became_peered":"2024-09-06T14:24:28.009420+0000","last_unstale":"2024-09-06T14:24:55.922923+0000","last_undegraded":"2024-09-06T14:24:55.922923+0000","last_fullsized":"2024-09-06T14:24:55.922923+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_clean_scrub_stamp":"2024-09-06T14:24:26.984416+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-08T01:05:42.106662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":30,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.448076+0000","last_change":"2024-09-06T14:24:25.998847+0000","last_active":"2024-09-06T14:24:55.448076+0000","last_peered":"2024-09-06T14:24:55.448076+0000","last_clean":"2024-09-06T14:24:55.448076+0000","last_became_active":"2024-09-06T14:24:25.998566+0000","last_became_peered":"2024-09-06T14:24:25.998566+0000","last_unstale":"2024-09-06T14:24:55.448076+0000","last_undegraded":"2024-09-06T14:24:55.448076+0000","last_fullsized":"2024-09-06T14:24:55.448076+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_clean_scrub_stamp":"2024-09-06T14:24:24.970251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T20:25:07.723740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":22,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.449936+0000","last_change":"2024-09-06T14:24:30.015643+0000","last_active":"2024-09-06T14:24:55.449936+0000","last_peered":"2024-09-06T14:24:55.449936+0000","last_clean":"2024-09-06T14:24:55.449936+0000","last_became_active":"2024-09-06T14:24:30.015322+0000","last_became_peered":"2024-09-06T14:24:30.015322+0000","last_unstale":"2024-09-06T14:24:55.449936+0000","last_undegraded":"2024-09-06T14:24:55.449936+0000","last_fullsized":"2024-09-06T14:24:55.449936+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_clean_scrub_stamp":"2024-09-06T14:24:28.992633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T16:13:02.441998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":18,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.448884+0000","last_change":"2024-09-06T14:24:32.029053+0000","last_active":"2024-09-06T14:24:55.448884+0000","last_peered":"2024-09-06T14:24:55.448884+0000","last_clean":"2024-09-06T14:24:55.448884+0000","last_became_active":"2024-09-06T14:24:32.028702+0000","last_became_peered":"2024-09-06T14:24:32.028702+0000","last_unstale":"2024-09-06T14:24:55.448884+0000","last_undegraded":"2024-09-06T14:24:55.448884+0000","last_fullsized":"2024-09-06T14:24:55.448884+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_clean_scrub_stamp":"2024-09-06T14:24:31.009248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T21:30:59.288934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"60'11","reported_seq":45,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.449720+0000","last_change":"2024-09-06T14:24:28.062589+0000","last_active":"2024-09-06T14:24:55.449720+0000","last_peered":"2024-09-06T14:24:55.449720+0000","last_clean":"2024-09-06T14:24:55.449720+0000","last_became_active":"2024-09-06T14:24:28.062290+0000","last_became_peered":"2024-09-06T14:24:28.062290+0000","last_unstale":"2024-09-06T14:24:55.449720+0000","last_undegraded":"2024-09-06T14:24:55.449720+0000","last_fullsized":"2024-09-06T14:24:55.449720+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_clean_scrub_stamp":"2024-09-06T14:24:26.984416+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T21:26:53.134114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":30,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.786196+0000","last_change":"2024-09-06T14:24:25.998515+0000","last_active":"2024-09-06T14:24:55.786196+0000","last_peered":"2024-09-06T14:24:55.786196+0000","last_clean":"2024-09-06T14:24:55.786196+0000","last_became_active":"2024-09-06T14:24:25.998174+0000","last_became_peered":"2024-09-06T14:24:25.998174+0000","last_unstale":"2024-09-06T14:24:55.786196+0000","last_undegraded":"2024-09-06T14:24:55.786196+0000","last_fullsized":"2024-09-06T14:24:55.786196+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_clean_scrub_stamp":"2024-09-06T14:24:24.970251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T21:18:35.673089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":22,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.446933+0000","last_change":"2024-09-06T14:24:30.015894+0000","last_active":"2024-09-06T14:24:55.446933+0000","last_peered":"2024-09-06T14:24:55.446933+0000","last_clean":"2024-09-06T14:24:55.446933+0000","last_became_active":"2024-09-06T14:24:30.015624+0000","last_became_peered":"2024-09-06T14:24:30.015624+0000","last_unstale":"2024-09-06T14:24:55.446933+0000","last_undegraded":"2024-09-06T14:24:55.446933+0000","last_fullsized":"2024-09-06T14:24:55.446933+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_clean_scrub_stamp":"2024-09-06T14:24:28.992633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T22:36:13.838324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":18,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.450088+0000","last_change":"2024-09-06T14:24:32.038464+0000","last_active":"2024-09-06T14:24:55.450088+0000","last_peered":"2024-09-06T14:24:55.450088+0000","last_clean":"2024-09-06T14:24:55.450088+0000","last_became_active":"2024-09-06T14:24:32.037605+0000","last_became_peered":"2024-09-06T14:24:32.037605+0000","last_unstale":"2024-09-06T14:24:55.450088+0000","last_undegraded":"2024-09-06T14:24:55.450088+0000","last_fullsized":"2024-09-06T14:24:55.450088+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_clean_scrub_stamp":"2024-09-06T14:24:31.009248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T22:48:10.569670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"60'4","reported_seq":32,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.449718+0000","last_change":"2024-09-06T14:24:28.015730+0000","last_active":"2024-09-06T14:24:55.449718+0000","last_peered":"2024-09-06T14:24:55.449718+0000","last_clean":"2024-09-06T14:24:55.449718+0000","last_became_active":"2024-09-06T14:24:28.015423+0000","last_became_peered":"2024-09-06T14:24:28.015423+0000","last_unstale":"2024-09-06T14:24:55.449718+0000","last_undegraded":"2024-09-06T14:24:55.449718+0000","last_fullsized":"2024-09-06T14:24:55.449718+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_clean_scrub_stamp":"2024-09-06T14:24:26.984416+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T14:54:45.334421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"53'1","reported_seq":36,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.449856+0000","last_change":"2024-09-06T14:24:26.002065+0000","last_active":"2024-09-06T14:24:55.449856+0000","last_peered":"2024-09-06T14:24:55.449856+0000","last_clean":"2024-09-06T14:24:55.449856+0000","last_became_active":"2024-09-06T14:24:25.999543+0000","last_became_peered":"2024-09-06T14:24:25.999543+0000","last_unstale":"2024-09-06T14:24:55.449856+0000","last_undegraded":"2024-09-06T14:24:55.449856+0000","last_fullsized":"2024-09-06T14:24:55.449856+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_clean_scrub_stamp":"2024-09-06T14:24:24.970251+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T20:12:56.043359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":22,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.921711+0000","last_change":"2024-09-06T14:24:30.014764+0000","last_active":"2024-09-06T14:24:55.921711+0000","last_peered":"2024-09-06T14:24:55.921711+0000","last_clean":"2024-09-06T14:24:55.921711+0000","last_became_active":"2024-09-06T14:24:30.014404+0000","last_became_peered":"2024-09-06T14:24:30.014404+0000","last_unstale":"2024-09-06T14:24:55.921711+0000","last_undegraded":"2024-09-06T14:24:55.921711+0000","last_fullsized":"2024-09-06T14:24:55.921711+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_clean_scrub_stamp":"2024-09-06T14:24:28.992633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T17:19:26.404116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":18,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.447142+0000","last_change":"2024-09-06T14:24:32.031767+0000","last_active":"2024-09-06T14:24:55.447142+0000","last_peered":"2024-09-06T14:24:55.447142+0000","last_clean":"2024-09-06T14:24:55.447142+0000","last_became_active":"2024-09-06T14:24:32.031430+0000","last_became_peered":"2024-09-06T14:24:32.031430+0000","last_unstale":"2024-09-06T14:24:55.447142+0000","last_undegraded":"2024-09-06T14:24:55.447142+0000","last_fullsized":"2024-09-06T14:24:55.447142+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_clean_scrub_stamp":"2024-09-06T14:24:31.009248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T19:35:08.072702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":18,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.922982+0000","last_change":"2024-09-06T14:24:32.033967+0000","last_active":"2024-09-06T14:24:55.922982+0000","last_peered":"2024-09-06T14:24:55.922982+0000","last_clean":"2024-09-06T14:24:55.922982+0000","last_became_active":"2024-09-06T14:24:32.033288+0000","last_became_peered":"2024-09-06T14:24:32.033288+0000","last_unstale":"2024-09-06T14:24:55.922982+0000","last_undegraded":"2024-09-06T14:24:55.922982+0000","last_fullsized":"2024-09-06T14:24:55.922982+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:31.009248+0000","last_clean_scrub_stamp":"2024-09-06T14:24:31.009248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T14:26:15.491944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":30,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.450772+0000","last_change":"2024-09-06T14:24:26.006072+0000","last_active":"2024-09-06T14:24:55.450772+0000","last_peered":"2024-09-06T14:24:55.450772+0000","last_clean":"2024-09-06T14:24:55.450772+0000","last_became_active":"2024-09-06T14:24:26.005840+0000","last_became_peered":"2024-09-06T14:24:26.005840+0000","last_unstale":"2024-09-06T14:24:55.450772+0000","last_undegraded":"2024-09-06T14:24:55.450772+0000","last_fullsized":"2024-09-06T14:24:55.450772+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:24.970251+0000","last_clean_scrub_stamp":"2024-09-06T14:24:24.970251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T19:45:19.119033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"60'11","reported_seq":45,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.922633+0000","last_change":"2024-09-06T14:24:28.008444+0000","last_active":"2024-09-06T14:24:55.922633+0000","last_peered":"2024-09-06T14:24:55.922633+0000","last_clean":"2024-09-06T14:24:55.922633+0000","last_became_active":"2024-09-06T14:24:28.008126+0000","last_became_peered":"2024-09-06T14:24:28.008126+0000","last_unstale":"2024-09-06T14:24:55.922633+0000","last_undegraded":"2024-09-06T14:24:55.922633+0000","last_fullsized":"2024-09-06T14:24:55.922633+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:26.984416+0000","last_clean_scrub_stamp":"2024-09-06T14:24:26.984416+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T22:28:51.014533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":22,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-06T14:24:55.450892+0000","last_change":"2024-09-06T14:24:30.017624+0000","last_active":"2024-09-06T14:24:55.450892+0000","last_peered":"2024-09-06T14:24:55.450892+0000","last_clean":"2024-09-06T14:24:55.450892+0000","last_became_active":"2024-09-06T14:24:30.017352+0000","last_became_peered":"2024-09-06T14:24:30.017352+0000","last_unstale":"2024-09-06T14:24:55.450892+0000","last_undegraded":"2024-09-06T14:24:55.450892+0000","last_fullsized":"2024-09-06T14:24:55.450892+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-06T14:24:28.992633+0000","last_clean_scrub_stamp":"2024-09-06T14:24:28.992633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-07T16:03:39.031274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":40,"num_read_kb":35,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":48,"seq":206158430221,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32380,"kb_used_data":1192,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712772,"statfs":{"total":95995035648,"available":95961878528,"internally_reserved":0,"allocated":1220608,"data_stored":822957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626450,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28200,"kb_used_data":1176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716952,"statfs":{"total":95995035648,"available":95966158848,"internally_reserved":0,"allocated":1204224,"data_stored":822308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822679,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31788,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713364,"statfs":{"total":95995035648,"available":95962484736,"internally_reserved":0,"allocated":614400,"data_stored":232856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849018908,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27656,"kb_used_data":628,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717496,"statfs":{"total":95995035648,"available":95966715904,"internally_reserved":0,"allocated":643072,"data_stored":232558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182434,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31820,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713332,"statfs":{"total":95995035648,"available":95962451968,"internally_reserved":0,"allocated":647168,"data_stored":233388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411366,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31780,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713372,"statfs":{"total":95995035648,"available":95962492928,"internally_reserved":0,"allocated":606208,"data_stored":231665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574891,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27692,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717460,"statfs":{"total":95995035648,"available":95966679040,"internally_reserved":0,"allocated":679936,"data_stored":233738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738417,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28224,"kb_used_data":1200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716928,"statfs":{"total":95995035648,"available":95966134272,"internally_reserved":0,"allocated":1228800,"data_stored":823659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-06T14:25:15.001 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-06T14:25:15.001 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-06T14:25:15.001 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-09-06T14:25:15.001 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph health --format=json 2024-09-06T14:25:15.274 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:25:15.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:14 smithi162 ceph-mon[33787]: pgmap v12: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:25:15.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:14 smithi162 ceph-mon[33787]: from='client.24620 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-06T14:25:15.394 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:14 smithi142 ceph-mon[29958]: pgmap v12: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:25:15.394 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:14 smithi142 ceph-mon[29958]: from='client.24620 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-06T14:25:15.394 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:14 smithi142 ceph-mon[37372]: pgmap v12: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:25:15.394 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:14 smithi142 ceph-mon[37372]: from='client.24620 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-06T14:25:16.016 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:15.921+0000 7f4cedfca640 1 -- 172.21.15.142:0/1374262312 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4ce81084f0 msgr2=0x7f4ce8108930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:25:16.017 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:15.921+0000 7f4cedfca640 1 --2- 172.21.15.142:0/1374262312 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4ce81084f0 0x7f4ce8108930 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f4cd0009900 tx=0x7f4cd002a410 comp rx=0 tx=0).stop 2024-09-06T14:25:16.017 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:15.921+0000 7f4cedfca640 1 -- 172.21.15.142:0/1374262312 shutdown_connections 2024-09-06T14:25:16.017 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:15.921+0000 7f4cedfca640 1 --2- 172.21.15.142:0/1374262312 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4ce8108e70 0x7f4ce810f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:16.017 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:15.921+0000 7f4cedfca640 1 --2- 172.21.15.142:0/1374262312 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4ce81084f0 0x7f4ce8108930 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:16.017 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:15.921+0000 7f4cedfca640 1 --2- 172.21.15.142:0/1374262312 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4ce8102ae0 0x7f4ce8102ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:16.017 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:15.921+0000 7f4cedfca640 1 -- 172.21.15.142:0/1374262312 >> 172.21.15.142:0/1374262312 conn(0x7f4ce80fe530 msgr2=0x7f4ce8100950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:25:16.017 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:15.921+0000 7f4cedfca640 1 -- 172.21.15.142:0/1374262312 shutdown_connections 2024-09-06T14:25:16.017 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:15.921+0000 7f4cedfca640 1 -- 172.21.15.142:0/1374262312 wait complete. 2024-09-06T14:25:16.017 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:15.921+0000 7f4cedfca640 1 Processor -- start 2024-09-06T14:25:16.017 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:15.922+0000 7f4cedfca640 1 -- start start 2024-09-06T14:25:16.017 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:15.922+0000 7f4cedfca640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4ce8102ae0 0x7f4ce8198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:25:16.017 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:15.922+0000 7f4cedfca640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4ce81084f0 0x7f4ce8198b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:25:16.017 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:15.922+0000 7f4cedfca640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4ce8108e70 0x7f4ce819df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:25:16.018 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:15.922+0000 7f4cedfca640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f4ce81121c0 con 0x7f4ce8102ae0 2024-09-06T14:25:16.018 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:15.922+0000 7f4cedfca640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f4ce8112040 con 0x7f4ce8108e70 2024-09-06T14:25:16.018 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:15.922+0000 7f4cedfca640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7f4ce8112340 con 0x7f4ce81084f0 2024-09-06T14:25:16.018 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:15.922+0000 7f4ce77fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4ce8102ae0 0x7f4ce8198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:25:16.018 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:15.922+0000 7f4cdffff640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4ce81084f0 0x7f4ce8198b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:25:16.018 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:15.922+0000 7f4cdffff640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4ce81084f0 0x7f4ce8198b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:35694/0 (socket says 172.21.15.142:35694) 2024-09-06T14:25:16.018 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:15.922+0000 7f4cdffff640 1 -- 172.21.15.142:0/1037659926 learned_addr learned my addr 172.21.15.142:0/1037659926 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:25:16.018 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:15.922+0000 7f4ce77fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4ce8102ae0 0x7f4ce8198630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:58298/0 (socket says 172.21.15.142:58298) 2024-09-06T14:25:16.018 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:15.922+0000 7f4ce7fff640 1 --2- 172.21.15.142:0/1037659926 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4ce8108e70 0x7f4ce819df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:25:16.018 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:15.922+0000 7f4ce77fe640 1 -- 172.21.15.142:0/1037659926 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4ce81084f0 msgr2=0x7f4ce8198b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:25:16.018 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:15.922+0000 7f4ce77fe640 1 --2- 172.21.15.142:0/1037659926 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4ce81084f0 0x7f4ce8198b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:16.018 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:15.922+0000 7f4ce77fe640 1 -- 172.21.15.142:0/1037659926 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4ce8108e70 msgr2=0x7f4ce819df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:25:16.018 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:15.922+0000 7f4ce77fe640 1 --2- 172.21.15.142:0/1037659926 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4ce8108e70 0x7f4ce819df10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:16.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:15.922+0000 7f4ce77fe640 1 -- 172.21.15.142:0/1037659926 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ce819e6e0 con 0x7f4ce8102ae0 2024-09-06T14:25:16.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:15.922+0000 7f4ce77fe640 1 --2- 172.21.15.142:0/1037659926 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4ce8102ae0 0x7f4ce8198630 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f4cd8009ce0 tx=0x7f4cd800b690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:25:16.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:15.923+0000 7f4ce57fa640 1 -- 172.21.15.142:0/1037659926 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cd8004020 con 0x7f4ce8102ae0 2024-09-06T14:25:16.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:15.923+0000 7f4ce57fa640 1 -- 172.21.15.142:0/1037659926 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7f4cd80035f0 con 0x7f4ce8102ae0 2024-09-06T14:25:16.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:15.923+0000 7f4cedfca640 1 -- 172.21.15.142:0/1037659926 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ce819e9d0 con 0x7f4ce8102ae0 2024-09-06T14:25:16.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:15.923+0000 7f4ce57fa640 1 -- 172.21.15.142:0/1037659926 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cd8019650 con 0x7f4ce8102ae0 2024-09-06T14:25:16.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:15.923+0000 7f4cedfca640 1 -- 172.21.15.142:0/1037659926 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ce80750e0 con 0x7f4ce8102ae0 2024-09-06T14:25:16.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:15.924+0000 7f4cedfca640 1 -- 172.21.15.142:0/1037659926 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ce8102ec0 con 0x7f4ce8102ae0 2024-09-06T14:25:16.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:15.925+0000 7f4ce57fa640 1 -- 172.21.15.142:0/1037659926 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 22) ==== 99844+0+0 (secure 0 0 0) 0x7f4cd8009310 con 0x7f4ce8102ae0 2024-09-06T14:25:16.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:15.926+0000 7f4ce57fa640 1 --2- 172.21.15.142:0/1037659926 >> v2:172.21.15.142:6800/2944545612 conn(0x7f4cd4077620 0x7f4cd4079ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:25:16.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:15.926+0000 7f4ce57fa640 1 -- 172.21.15.142:0/1037659926 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 6043+0+0 (secure 0 0 0) 0x7f4cd809d1a0 con 0x7f4ce8102ae0 2024-09-06T14:25:16.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:15.926+0000 7f4cdffff640 1 --2- 172.21.15.142:0/1037659926 >> v2:172.21.15.142:6800/2944545612 conn(0x7f4cd4077620 0x7f4cd4079ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:25:16.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:15.926+0000 7f4cdffff640 1 --2- 172.21.15.142:0/1037659926 >> v2:172.21.15.142:6800/2944545612 conn(0x7f4cd4077620 0x7f4cd4079ae0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f4cd0031440 tx=0x7f4cd00058d0 comp rx=0 tx=0).ready entity=mgr.14601 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:25:16.020 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:15.930+0000 7f4ce57fa640 1 -- 172.21.15.142:0/1037659926 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7f4cd806a410 con 0x7f4ce8102ae0 2024-09-06T14:25:16.126 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:16.129+0000 7f4cedfca640 1 -- 172.21.15.142:0/1037659926 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f4ce810ac50 con 0x7f4ce8102ae0 2024-09-06T14:25:16.126 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:16.129+0000 7f4ce57fa640 1 -- 172.21.15.142:0/1037659926 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+212 (secure 0 0 0) 0x7f4cd806ee30 con 0x7f4ce8102ae0 2024-09-06T14:25:16.126 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:25:16.126 INFO:teuthology.orchestra.run.smithi142.stdout:{"status":"HEALTH_WARN","checks":{"MGR_MODULE_DEPENDENCY":{"severity":"HEALTH_WARN","summary":{"message":"Module 'volumes' has failed dependency: No module named 'ceph.fs'","count":1},"muted":false}},"mutes":[]} 2024-09-06T14:25:16.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:16.131+0000 7f4cedfca640 1 -- 172.21.15.142:0/1037659926 >> v2:172.21.15.142:6800/2944545612 conn(0x7f4cd4077620 msgr2=0x7f4cd4079ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:25:16.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:16.131+0000 7f4cedfca640 1 --2- 172.21.15.142:0/1037659926 >> v2:172.21.15.142:6800/2944545612 conn(0x7f4cd4077620 0x7f4cd4079ae0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f4cd0031440 tx=0x7f4cd00058d0 comp rx=0 tx=0).stop 2024-09-06T14:25:16.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:16.131+0000 7f4cedfca640 1 -- 172.21.15.142:0/1037659926 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4ce8102ae0 msgr2=0x7f4ce8198630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:25:16.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:16.131+0000 7f4cedfca640 1 --2- 172.21.15.142:0/1037659926 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4ce8102ae0 0x7f4ce8198630 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f4cd8009ce0 tx=0x7f4cd800b690 comp rx=0 tx=0).stop 2024-09-06T14:25:16.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:16.131+0000 7f4cedfca640 1 -- 172.21.15.142:0/1037659926 shutdown_connections 2024-09-06T14:25:16.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:16.131+0000 7f4cedfca640 1 --2- 172.21.15.142:0/1037659926 >> v2:172.21.15.142:6800/2944545612 conn(0x7f4cd4077620 0x7f4cd4079ae0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:16.129 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:16.131+0000 7f4cedfca640 1 --2- 172.21.15.142:0/1037659926 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4ce8108e70 0x7f4ce819df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:16.129 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:16.131+0000 7f4cedfca640 1 --2- 172.21.15.142:0/1037659926 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4ce81084f0 0x7f4ce8198b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:16.129 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:16.131+0000 7f4cedfca640 1 --2- 172.21.15.142:0/1037659926 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4ce8102ae0 0x7f4ce8198630 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:16.129 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:16.131+0000 7f4cedfca640 1 -- 172.21.15.142:0/1037659926 >> 172.21.15.142:0/1037659926 conn(0x7f4ce80fe530 msgr2=0x7f4ce8100920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:25:16.129 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:16.131+0000 7f4cedfca640 1 -- 172.21.15.142:0/1037659926 shutdown_connections 2024-09-06T14:25:16.129 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:16.131+0000 7f4cedfca640 1 -- 172.21.15.142:0/1037659926 wait complete. 2024-09-06T14:25:17.171 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:16 smithi142 ceph-mon[29958]: pgmap v13: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:25:17.171 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:16 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/1037659926' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:25:17.171 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:16 smithi142 ceph-mon[37372]: pgmap v13: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:25:17.171 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:16 smithi142 ceph-mon[37372]: from='client.? 172.21.15.142:0/1037659926' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:25:17.297 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 06 14:25:17 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-iscsi-iscsi-a[59853]: debug there is no tcmu-runner data available 2024-09-06T14:25:17.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:16 smithi162 ceph-mon[33787]: pgmap v13: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:25:17.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:16 smithi162 ceph-mon[33787]: from='client.? 172.21.15.142:0/1037659926' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:25:18.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:17 smithi162 ceph-mon[33787]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-06T14:25:18.459 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:17 smithi142 ceph-mon[29958]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-06T14:25:18.459 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:17 smithi142 ceph-mon[37372]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-06T14:25:19.087 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:18 smithi142 ceph-mon[29958]: pgmap v14: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:25:19.087 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:18 smithi142 ceph-mon[37372]: pgmap v14: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:25:19.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:18 smithi162 ceph-mon[33787]: pgmap v14: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:25:20.554 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph health --format=json 2024-09-06T14:25:21.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:20 smithi142 ceph-mon[29958]: pgmap v15: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:25:21.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:20 smithi142 ceph-mon[37372]: pgmap v15: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:25:21.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:20 smithi162 ceph-mon[33787]: pgmap v15: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:25:21.878 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:25:22.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:22.383+0000 7f09b02a3640 1 -- 172.21.15.142:0/878853109 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f09a81042c0 msgr2=0x7f09a810c8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:25:22.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:22.383+0000 7f09b02a3640 1 --2- 172.21.15.142:0/878853109 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f09a81042c0 0x7f09a810c8a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0998009860 tx=0x7f099802a340 comp rx=0 tx=0).stop 2024-09-06T14:25:22.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:22.383+0000 7f09b02a3640 1 -- 172.21.15.142:0/878853109 shutdown_connections 2024-09-06T14:25:22.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:22.383+0000 7f09b02a3640 1 --2- 172.21.15.142:0/878853109 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f09a810ce70 0x7f09a810f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:22.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:22.383+0000 7f09b02a3640 1 --2- 172.21.15.142:0/878853109 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f09a81042c0 0x7f09a810c8a0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:22.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:22.383+0000 7f09b02a3640 1 --2- 172.21.15.142:0/878853109 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f09a81038c0 0x7f09a8103ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:22.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:22.383+0000 7f09b02a3640 1 -- 172.21.15.142:0/878853109 >> 172.21.15.142:0/878853109 conn(0x7f09a80fd430 msgr2=0x7f09a80ff850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:25:22.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:22.383+0000 7f09b02a3640 1 -- 172.21.15.142:0/878853109 shutdown_connections 2024-09-06T14:25:22.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:22.383+0000 7f09b02a3640 1 -- 172.21.15.142:0/878853109 wait complete. 2024-09-06T14:25:22.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:22.383+0000 7f09b02a3640 1 Processor -- start 2024-09-06T14:25:22.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:22.383+0000 7f09b02a3640 1 -- start start 2024-09-06T14:25:22.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:22.384+0000 7f09b02a3640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f09a81038c0 0x7f09a8198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:25:22.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:22.384+0000 7f09b02a3640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f09a81042c0 0x7f09a8198c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:25:22.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:22.384+0000 7f09b02a3640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f09a810ce70 0x7f09a819dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:25:22.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:22.384+0000 7f09b02a3640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f09a81121a0 con 0x7f09a81038c0 2024-09-06T14:25:22.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:22.384+0000 7f09b02a3640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f09a8112020 con 0x7f09a81042c0 2024-09-06T14:25:22.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:22.384+0000 7f09b02a3640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7f09a8112320 con 0x7f09a810ce70 2024-09-06T14:25:22.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:22.384+0000 7f09ae03f640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f09a81038c0 0x7f09a8198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:25:22.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:22.384+0000 7f09ae03f640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f09a81038c0 0x7f09a8198700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:46968/0 (socket says 172.21.15.142:46968) 2024-09-06T14:25:22.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:22.384+0000 7f09ae03f640 1 -- 172.21.15.142:0/2107792729 learned_addr learned my addr 172.21.15.142:0/2107792729 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:25:22.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:22.384+0000 7f09ae840640 1 --2- 172.21.15.142:0/2107792729 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f09a810ce70 0x7f09a819dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:25:22.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:22.384+0000 7f09ad83e640 1 --2- 172.21.15.142:0/2107792729 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f09a81042c0 0x7f09a8198c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:25:22.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:22.384+0000 7f09ae03f640 1 -- 172.21.15.142:0/2107792729 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f09a810ce70 msgr2=0x7f09a819dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:25:22.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:22.384+0000 7f09ae03f640 1 --2- 172.21.15.142:0/2107792729 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f09a810ce70 0x7f09a819dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:22.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:22.384+0000 7f09ae03f640 1 -- 172.21.15.142:0/2107792729 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f09a81042c0 msgr2=0x7f09a8198c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:25:22.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:22.384+0000 7f09ae03f640 1 --2- 172.21.15.142:0/2107792729 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f09a81042c0 0x7f09a8198c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:22.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:22.384+0000 7f09ae03f640 1 -- 172.21.15.142:0/2107792729 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09a819e7b0 con 0x7f09a81038c0 2024-09-06T14:25:22.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:22.384+0000 7f09ae03f640 1 --2- 172.21.15.142:0/2107792729 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f09a81038c0 0x7f09a8198700 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f09a400d770 tx=0x7f09a400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:25:22.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:22.384+0000 7f099f7fe640 1 -- 172.21.15.142:0/2107792729 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09a4017070 con 0x7f09a81038c0 2024-09-06T14:25:22.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:22.385+0000 7f09b02a3640 1 -- 172.21.15.142:0/2107792729 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09a819eaa0 con 0x7f09a81038c0 2024-09-06T14:25:22.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:22.385+0000 7f099f7fe640 1 -- 172.21.15.142:0/2107792729 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7f09a40054a0 con 0x7f09a81038c0 2024-09-06T14:25:22.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:22.385+0000 7f099f7fe640 1 -- 172.21.15.142:0/2107792729 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09a4004c40 con 0x7f09a81038c0 2024-09-06T14:25:22.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:22.385+0000 7f09b02a3640 1 -- 172.21.15.142:0/2107792729 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09a8101620 con 0x7f09a81038c0 2024-09-06T14:25:22.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:22.386+0000 7f09b02a3640 1 -- 172.21.15.142:0/2107792729 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09a8103ca0 con 0x7f09a81038c0 2024-09-06T14:25:22.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:22.386+0000 7f099f7fe640 1 -- 172.21.15.142:0/2107792729 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 22) ==== 99844+0+0 (secure 0 0 0) 0x7f09a4004de0 con 0x7f09a81038c0 2024-09-06T14:25:22.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:22.387+0000 7f099f7fe640 1 --2- 172.21.15.142:0/2107792729 >> v2:172.21.15.142:6800/2944545612 conn(0x7f0994077620 0x7f0994079ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:25:22.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:22.387+0000 7f099f7fe640 1 -- 172.21.15.142:0/2107792729 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 6043+0+0 (secure 0 0 0) 0x7f09a40993e0 con 0x7f09a81038c0 2024-09-06T14:25:22.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:22.387+0000 7f09ad83e640 1 --2- 172.21.15.142:0/2107792729 >> v2:172.21.15.142:6800/2944545612 conn(0x7f0994077620 0x7f0994079ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:25:22.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:22.388+0000 7f09ad83e640 1 --2- 172.21.15.142:0/2107792729 >> v2:172.21.15.142:6800/2944545612 conn(0x7f0994077620 0x7f0994079ae0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f099802a930 tx=0x7f09980058d0 comp rx=0 tx=0).ready entity=mgr.14601 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:25:22.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:22.391+0000 7f099f7fe640 1 -- 172.21.15.142:0/2107792729 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7f09a4066670 con 0x7f09a81038c0 2024-09-06T14:25:22.569 INFO:journalctl@ceph.alertmanager.a.smithi142.stdout:Sep 06 14:25:22 smithi142 systemd[1]: Starting Ceph alertmanager.a for ce32462e-6c5a-11ef-bce4-c7b262605968... 2024-09-06T14:25:22.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:22.586+0000 7f09b02a3640 1 -- 172.21.15.142:0/2107792729 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f09a81075f0 con 0x7f09a81038c0 2024-09-06T14:25:22.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:22.586+0000 7f099f7fe640 1 -- 172.21.15.142:0/2107792729 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+212 (secure 0 0 0) 0x7f09a406b090 con 0x7f09a81038c0 2024-09-06T14:25:22.583 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:25:22.583 INFO:teuthology.orchestra.run.smithi142.stdout:{"status":"HEALTH_WARN","checks":{"MGR_MODULE_DEPENDENCY":{"severity":"HEALTH_WARN","summary":{"message":"Module 'volumes' has failed dependency: No module named 'ceph.fs'","count":1},"muted":false}},"mutes":[]} 2024-09-06T14:25:22.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:22.588+0000 7f09b02a3640 1 -- 172.21.15.142:0/2107792729 >> v2:172.21.15.142:6800/2944545612 conn(0x7f0994077620 msgr2=0x7f0994079ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:25:22.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:22.588+0000 7f09b02a3640 1 --2- 172.21.15.142:0/2107792729 >> v2:172.21.15.142:6800/2944545612 conn(0x7f0994077620 0x7f0994079ae0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f099802a930 tx=0x7f09980058d0 comp rx=0 tx=0).stop 2024-09-06T14:25:22.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:22.588+0000 7f09b02a3640 1 -- 172.21.15.142:0/2107792729 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f09a81038c0 msgr2=0x7f09a8198700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:25:22.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:22.588+0000 7f09b02a3640 1 --2- 172.21.15.142:0/2107792729 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f09a81038c0 0x7f09a8198700 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f09a400d770 tx=0x7f09a400dc30 comp rx=0 tx=0).stop 2024-09-06T14:25:22.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:22.588+0000 7f09b02a3640 1 -- 172.21.15.142:0/2107792729 shutdown_connections 2024-09-06T14:25:22.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:22.588+0000 7f09b02a3640 1 --2- 172.21.15.142:0/2107792729 >> v2:172.21.15.142:6800/2944545612 conn(0x7f0994077620 0x7f0994079ae0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:22.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:22.588+0000 7f09b02a3640 1 --2- 172.21.15.142:0/2107792729 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f09a810ce70 0x7f09a819dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:22.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:22.588+0000 7f09b02a3640 1 --2- 172.21.15.142:0/2107792729 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f09a81042c0 0x7f09a8198c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:22.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:22.588+0000 7f09b02a3640 1 --2- 172.21.15.142:0/2107792729 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f09a81038c0 0x7f09a8198700 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:22.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:22.588+0000 7f09b02a3640 1 -- 172.21.15.142:0/2107792729 >> 172.21.15.142:0/2107792729 conn(0x7f09a80fd430 msgr2=0x7f09a81047c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:25:22.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:22.588+0000 7f09b02a3640 1 -- 172.21.15.142:0/2107792729 shutdown_connections 2024-09-06T14:25:22.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:22.588+0000 7f09b02a3640 1 -- 172.21.15.142:0/2107792729 wait complete. 2024-09-06T14:25:23.187 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:22 smithi142 ceph-mon[29958]: pgmap v16: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:25:23.187 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:22 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/2107792729' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:25:23.187 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:22 smithi142 ceph-mon[37372]: pgmap v16: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:25:23.187 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:22 smithi142 ceph-mon[37372]: from='client.? 172.21.15.142:0/2107792729' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:25:23.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:22 smithi162 ceph-mon[33787]: pgmap v16: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:25:23.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:22 smithi162 ceph-mon[33787]: from='client.? 172.21.15.142:0/2107792729' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:25:23.704 INFO:journalctl@ceph.alertmanager.a.smithi142.stdout:Sep 06 14:25:23 smithi142 podman[70800]: 2024-09-06 14:25:23.450329429 +0000 UTC m=+0.111776494 volume create 49ed157c762f9b402cee411a43398b4273cd07bf071eaebd683632a5a3d253fb 2024-09-06T14:25:23.704 INFO:journalctl@ceph.alertmanager.a.smithi142.stdout:Sep 06 14:25:23 smithi142 podman[70800]: 2024-09-06 14:25:23.354433171 +0000 UTC m=+0.015880205 image pull 11f11916f8cdfb17c1da74b20deb2075d778357b4d60474d82453303f8d461f5 quay.io/prometheus/alertmanager:v0.27.0 2024-09-06T14:25:23.704 INFO:journalctl@ceph.alertmanager.a.smithi142.stdout:Sep 06 14:25:23 smithi142 podman[70800]: 2024-09-06 14:25:23.534787004 +0000 UTC m=+0.196234085 container create 9bea55ee93597a4f79718b69728fa4198413da559e7357778dd9aa67cae822e3 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-06T14:25:24.209 INFO:journalctl@ceph.alertmanager.a.smithi142.stdout:Sep 06 14:25:23 smithi142 podman[70800]: 2024-09-06 14:25:23.837742825 +0000 UTC m=+0.499189851 container init 9bea55ee93597a4f79718b69728fa4198413da559e7357778dd9aa67cae822e3 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-06T14:25:24.209 INFO:journalctl@ceph.alertmanager.a.smithi142.stdout:Sep 06 14:25:23 smithi142 podman[70800]: 2024-09-06 14:25:23.840647195 +0000 UTC m=+0.502094220 container start 9bea55ee93597a4f79718b69728fa4198413da559e7357778dd9aa67cae822e3 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-06T14:25:24.210 INFO:journalctl@ceph.alertmanager.a.smithi142.stdout:Sep 06 14:25:23 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-alertmanager-a[70815]: ts=2024-09-06T14:25:23.851Z caller=main.go:181 level=info msg="Starting Alertmanager" version="(version=0.27.0, branch=HEAD, revision=0aa3c2aad14cff039931923ab16b26b7481783b5)" 2024-09-06T14:25:24.210 INFO:journalctl@ceph.alertmanager.a.smithi142.stdout:Sep 06 14:25:23 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-alertmanager-a[70815]: ts=2024-09-06T14:25:23.851Z caller=main.go:182 level=info build_context="(go=go1.21.7, platform=linux/amd64, user=root@22cd11f671e9, date=20240228-11:51:20, tags=netgo)" 2024-09-06T14:25:24.210 INFO:journalctl@ceph.alertmanager.a.smithi142.stdout:Sep 06 14:25:23 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-alertmanager-a[70815]: ts=2024-09-06T14:25:23.852Z caller=cluster.go:186 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.142 port=9094 2024-09-06T14:25:24.210 INFO:journalctl@ceph.alertmanager.a.smithi142.stdout:Sep 06 14:25:23 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-alertmanager-a[70815]: ts=2024-09-06T14:25:23.853Z caller=cluster.go:683 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-09-06T14:25:24.210 INFO:journalctl@ceph.alertmanager.a.smithi142.stdout:Sep 06 14:25:23 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-alertmanager-a[70815]: ts=2024-09-06T14:25:23.875Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-06T14:25:24.210 INFO:journalctl@ceph.alertmanager.a.smithi142.stdout:Sep 06 14:25:23 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-alertmanager-a[70815]: ts=2024-09-06T14:25:23.875Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-06T14:25:24.210 INFO:journalctl@ceph.alertmanager.a.smithi142.stdout:Sep 06 14:25:23 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-alertmanager-a[70815]: ts=2024-09-06T14:25:23.877Z caller=tls_config.go:313 level=info msg="Listening on" address=[::]:9093 2024-09-06T14:25:24.210 INFO:journalctl@ceph.alertmanager.a.smithi142.stdout:Sep 06 14:25:23 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-alertmanager-a[70815]: ts=2024-09-06T14:25:23.877Z caller=tls_config.go:316 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-09-06T14:25:24.210 INFO:journalctl@ceph.alertmanager.a.smithi142.stdout:Sep 06 14:25:23 smithi142 bash[70800]: 9bea55ee93597a4f79718b69728fa4198413da559e7357778dd9aa67cae822e3 2024-09-06T14:25:24.210 INFO:journalctl@ceph.alertmanager.a.smithi142.stdout:Sep 06 14:25:23 smithi142 systemd[1]: Started Ceph alertmanager.a for ce32462e-6c5a-11ef-bce4-c7b262605968. 2024-09-06T14:25:24.708 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:25:24 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: ::ffff:172.21.15.162 - - [06/Sep/2024:14:25:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-06T14:25:25.209 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:25:24 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: [06/Sep/2024:14:25:24] ENGINE Bus STOPPING 2024-09-06T14:25:25.209 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:25:24 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: [06/Sep/2024:14:25:24] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-06T14:25:25.209 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:25:24 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: [06/Sep/2024:14:25:24] ENGINE Bus STOPPED 2024-09-06T14:25:25.209 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:25:24 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: [06/Sep/2024:14:25:24] ENGINE Bus STARTING 2024-09-06T14:25:25.209 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:25:25 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: [06/Sep/2024:14:25:25] ENGINE Serving on http://:::9283 2024-09-06T14:25:25.209 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:25:25 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: [06/Sep/2024:14:25:25] ENGINE Bus STARTED 2024-09-06T14:25:25.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:24 smithi142 ceph-mon[37372]: pgmap v17: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:25:25.210 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:24 smithi142 ceph-mon[37372]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:25.210 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:24 smithi142 ceph-mon[37372]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:25.210 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:24 smithi142 ceph-mon[37372]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:25.210 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:24 smithi142 ceph-mon[37372]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:25.210 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:24 smithi142 ceph-mon[37372]: Regenerating cephadm self-signed grafana TLS certificates 2024-09-06T14:25:25.210 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:24 smithi142 ceph-mon[37372]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:25.210 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:24 smithi142 ceph-mon[37372]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:25.210 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:24 smithi142 ceph-mon[37372]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"} : dispatch 2024-09-06T14:25:25.210 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:24 smithi142 ceph-mon[37372]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:25.210 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:24 smithi142 ceph-mon[37372]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-06T14:25:25.210 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:24 smithi142 ceph-mon[29958]: pgmap v17: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:25:25.211 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:24 smithi142 ceph-mon[29958]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:25.211 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:24 smithi142 ceph-mon[29958]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:25.211 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:24 smithi142 ceph-mon[29958]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:25.211 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:24 smithi142 ceph-mon[29958]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:25.211 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:24 smithi142 ceph-mon[29958]: Regenerating cephadm self-signed grafana TLS certificates 2024-09-06T14:25:25.211 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:24 smithi142 ceph-mon[29958]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:25.211 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:24 smithi142 ceph-mon[29958]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:25.211 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:24 smithi142 ceph-mon[29958]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"} : dispatch 2024-09-06T14:25:25.211 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:24 smithi142 ceph-mon[29958]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:25.211 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:24 smithi142 ceph-mon[29958]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-06T14:25:25.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:24 smithi162 ceph-mon[33787]: pgmap v17: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:25:25.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:24 smithi162 ceph-mon[33787]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:25.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:24 smithi162 ceph-mon[33787]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:25.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:24 smithi162 ceph-mon[33787]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:25.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:24 smithi162 ceph-mon[33787]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:25.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:24 smithi162 ceph-mon[33787]: Regenerating cephadm self-signed grafana TLS certificates 2024-09-06T14:25:25.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:24 smithi162 ceph-mon[33787]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:25.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:24 smithi162 ceph-mon[33787]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:25.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:24 smithi162 ceph-mon[33787]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"} : dispatch 2024-09-06T14:25:25.298 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:24 smithi162 ceph-mon[33787]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:25.298 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:24 smithi162 ceph-mon[33787]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-06T14:25:26.209 INFO:journalctl@ceph.alertmanager.a.smithi142.stdout:Sep 06 14:25:25 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-alertmanager-a[70815]: ts=2024-09-06T14:25:25.853Z caller=cluster.go:708 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000296352s 2024-09-06T14:25:26.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:25 smithi142 ceph-mon[29958]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-06T14:25:26.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:25 smithi142 ceph-mon[29958]: Deploying daemon grafana.a on smithi162 2024-09-06T14:25:26.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:25 smithi142 ceph-mon[29958]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-06T14:25:26.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:25 smithi142 ceph-mon[29958]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:26.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:25 smithi142 ceph-mon[29958]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-06T14:25:26.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:25 smithi142 ceph-mon[37372]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-06T14:25:26.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:25 smithi142 ceph-mon[37372]: Deploying daemon grafana.a on smithi162 2024-09-06T14:25:26.210 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:25 smithi142 ceph-mon[37372]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-06T14:25:26.210 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:25 smithi142 ceph-mon[37372]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:26.210 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:25 smithi142 ceph-mon[37372]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-06T14:25:26.219 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph health --format=json 2024-09-06T14:25:26.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:25 smithi162 ceph-mon[33787]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-06T14:25:26.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:25 smithi162 ceph-mon[33787]: Deploying daemon grafana.a on smithi162 2024-09-06T14:25:26.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:25 smithi162 ceph-mon[33787]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-06T14:25:26.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:25 smithi162 ceph-mon[33787]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:26.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:25 smithi162 ceph-mon[33787]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-06T14:25:26.480 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:25:26.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:26.960+0000 7f2fa64f5640 1 -- 172.21.15.142:0/939293076 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2fa0107d90 msgr2=0x7f2fa010e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:25:26.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:26.960+0000 7f2fa64f5640 1 --2- 172.21.15.142:0/939293076 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2fa0107d90 0x7f2fa010e590 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2f98009ca0 tx=0x7f2f9802a8c0 comp rx=0 tx=0).stop 2024-09-06T14:25:26.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:26.960+0000 7f2fa64f5640 1 -- 172.21.15.142:0/939293076 shutdown_connections 2024-09-06T14:25:26.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:26.960+0000 7f2fa64f5640 1 --2- 172.21.15.142:0/939293076 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2fa0107d90 0x7f2fa010e590 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:26.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:26.960+0000 7f2fa64f5640 1 --2- 172.21.15.142:0/939293076 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2fa0107410 0x7f2fa0107850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:26.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:26.960+0000 7f2fa64f5640 1 --2- 172.21.15.142:0/939293076 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2fa0101a00 0x7f2fa0101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:26.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:26.960+0000 7f2fa64f5640 1 -- 172.21.15.142:0/939293076 >> 172.21.15.142:0/939293076 conn(0x7f2fa00fd470 msgr2=0x7f2fa00ff890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:25:26.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:26.960+0000 7f2fa64f5640 1 -- 172.21.15.142:0/939293076 shutdown_connections 2024-09-06T14:25:26.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:26.960+0000 7f2fa64f5640 1 -- 172.21.15.142:0/939293076 wait complete. 2024-09-06T14:25:26.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:26.961+0000 7f2fa64f5640 1 Processor -- start 2024-09-06T14:25:26.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:26.961+0000 7f2fa64f5640 1 -- start start 2024-09-06T14:25:26.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:26.961+0000 7f2fa64f5640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2fa0101a00 0x7f2fa0198650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:25:26.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:26.961+0000 7f2fa64f5640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2fa0107410 0x7f2fa0198b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:25:26.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:26.961+0000 7f2fa64f5640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2fa0107d90 0x7f2fa019df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:25:26.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:26.961+0000 7f2fa64f5640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f2fa0112260 con 0x7f2fa0107410 2024-09-06T14:25:26.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:26.961+0000 7f2fa64f5640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f2fa01120e0 con 0x7f2fa0107d90 2024-09-06T14:25:26.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:26.962+0000 7f2fa64f5640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7f2fa01123e0 con 0x7f2fa0101a00 2024-09-06T14:25:26.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:26.962+0000 7f2f9ffff640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2fa0101a00 0x7f2fa0198650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:25:26.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:26.962+0000 7f2f9f7fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2fa0107410 0x7f2fa0198b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:25:26.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:26.962+0000 7f2f9ffff640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2fa0101a00 0x7f2fa0198650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:56734/0 (socket says 172.21.15.142:56734) 2024-09-06T14:25:26.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:26.962+0000 7f2f9ffff640 1 -- 172.21.15.142:0/3325415424 learned_addr learned my addr 172.21.15.142:0/3325415424 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:25:26.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:26.962+0000 7f2fa4a92640 1 --2- 172.21.15.142:0/3325415424 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2fa0107d90 0x7f2fa019df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:25:26.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:26.962+0000 7f2f9f7fe640 1 -- 172.21.15.142:0/3325415424 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2fa0101a00 msgr2=0x7f2fa0198650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:25:26.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:26.962+0000 7f2f9f7fe640 1 --2- 172.21.15.142:0/3325415424 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2fa0101a00 0x7f2fa0198650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:26.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:26.962+0000 7f2f9f7fe640 1 -- 172.21.15.142:0/3325415424 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2fa0107d90 msgr2=0x7f2fa019df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:25:26.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:26.962+0000 7f2f9f7fe640 1 --2- 172.21.15.142:0/3325415424 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2fa0107d90 0x7f2fa019df30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:26.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:26.962+0000 7f2f9f7fe640 1 -- 172.21.15.142:0/3325415424 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fa019e700 con 0x7f2fa0107410 2024-09-06T14:25:26.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:26.962+0000 7f2f9ffff640 1 --2- 172.21.15.142:0/3325415424 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2fa0101a00 0x7f2fa0198650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-06T14:25:26.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:26.962+0000 7f2f9f7fe640 1 --2- 172.21.15.142:0/3325415424 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2fa0107410 0x7f2fa0198b90 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f2f9400e9f0 tx=0x7f2f9400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:25:26.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:26.962+0000 7f2f9d7fa640 1 -- 172.21.15.142:0/3325415424 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f9400cd80 con 0x7f2fa0107410 2024-09-06T14:25:26.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:26.962+0000 7f2f9d7fa640 1 -- 172.21.15.142:0/3325415424 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7f2f940035f0 con 0x7f2fa0107410 2024-09-06T14:25:26.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:26.962+0000 7f2fa64f5640 1 -- 172.21.15.142:0/3325415424 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2fa019e9f0 con 0x7f2fa0107410 2024-09-06T14:25:26.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:26.962+0000 7f2f9d7fa640 1 -- 172.21.15.142:0/3325415424 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f940044a0 con 0x7f2fa0107410 2024-09-06T14:25:26.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:26.962+0000 7f2fa64f5640 1 -- 172.21.15.142:0/3325415424 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2fa01a5280 con 0x7f2fa0107410 2024-09-06T14:25:26.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:26.963+0000 7f2fa64f5640 1 -- 172.21.15.142:0/3325415424 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2fa0101de0 con 0x7f2fa0107410 2024-09-06T14:25:26.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:26.964+0000 7f2f9d7fa640 1 -- 172.21.15.142:0/3325415424 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 22) ==== 99844+0+0 (secure 0 0 0) 0x7f2f94004020 con 0x7f2fa0107410 2024-09-06T14:25:26.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:26.964+0000 7f2f9d7fa640 1 --2- 172.21.15.142:0/3325415424 >> v2:172.21.15.142:6800/2944545612 conn(0x7f2f88077620 0x7f2f88079ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:25:26.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:26.964+0000 7f2f9d7fa640 1 -- 172.21.15.142:0/3325415424 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 6043+0+0 (secure 0 0 0) 0x7f2f94099b10 con 0x7f2fa0107410 2024-09-06T14:25:26.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:26.964+0000 7f2f9ffff640 1 --2- 172.21.15.142:0/3325415424 >> v2:172.21.15.142:6800/2944545612 conn(0x7f2f88077620 0x7f2f88079ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:25:26.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:26.965+0000 7f2f9ffff640 1 --2- 172.21.15.142:0/3325415424 >> v2:172.21.15.142:6800/2944545612 conn(0x7f2f88077620 0x7f2f88079ae0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f2f90011500 tx=0x7f2f90005f90 comp rx=0 tx=0).ready entity=mgr.14601 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:25:26.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:26.968+0000 7f2f9d7fa640 1 -- 172.21.15.142:0/3325415424 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7f2f94066e20 con 0x7f2fa0107410 2024-09-06T14:25:27.155 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:27.158+0000 7f2fa64f5640 1 -- 172.21.15.142:0/3325415424 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f2fa01a5450 con 0x7f2fa0107410 2024-09-06T14:25:27.156 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:26 smithi142 ceph-mon[29958]: pgmap v18: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:25:27.156 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:26 smithi142 ceph-mon[37372]: pgmap v18: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:25:27.156 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:27.159+0000 7f2f9d7fa640 1 -- 172.21.15.142:0/3325415424 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+212 (secure 0 0 0) 0x7f2f9406b160 con 0x7f2fa0107410 2024-09-06T14:25:27.156 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:25:27.156 INFO:teuthology.orchestra.run.smithi142.stdout:{"status":"HEALTH_WARN","checks":{"MGR_MODULE_DEPENDENCY":{"severity":"HEALTH_WARN","summary":{"message":"Module 'volumes' has failed dependency: No module named 'ceph.fs'","count":1},"muted":false}},"mutes":[]} 2024-09-06T14:25:27.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:27.161+0000 7f2fa64f5640 1 -- 172.21.15.142:0/3325415424 >> v2:172.21.15.142:6800/2944545612 conn(0x7f2f88077620 msgr2=0x7f2f88079ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:25:27.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:27.161+0000 7f2fa64f5640 1 --2- 172.21.15.142:0/3325415424 >> v2:172.21.15.142:6800/2944545612 conn(0x7f2f88077620 0x7f2f88079ae0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f2f90011500 tx=0x7f2f90005f90 comp rx=0 tx=0).stop 2024-09-06T14:25:27.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:27.161+0000 7f2fa64f5640 1 -- 172.21.15.142:0/3325415424 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2fa0107410 msgr2=0x7f2fa0198b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:25:27.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:27.161+0000 7f2fa64f5640 1 --2- 172.21.15.142:0/3325415424 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2fa0107410 0x7f2fa0198b90 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f2f9400e9f0 tx=0x7f2f9400eeb0 comp rx=0 tx=0).stop 2024-09-06T14:25:27.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:27.161+0000 7f2fa64f5640 1 -- 172.21.15.142:0/3325415424 shutdown_connections 2024-09-06T14:25:27.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:27.161+0000 7f2fa64f5640 1 --2- 172.21.15.142:0/3325415424 >> v2:172.21.15.142:6800/2944545612 conn(0x7f2f88077620 0x7f2f88079ae0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:27.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:27.161+0000 7f2fa64f5640 1 --2- 172.21.15.142:0/3325415424 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2fa0107d90 0x7f2fa019df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:27.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:27.161+0000 7f2fa64f5640 1 --2- 172.21.15.142:0/3325415424 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2fa0107410 0x7f2fa0198b90 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:27.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:27.161+0000 7f2fa64f5640 1 --2- 172.21.15.142:0/3325415424 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2fa0101a00 0x7f2fa0198650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:27.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:27.161+0000 7f2fa64f5640 1 -- 172.21.15.142:0/3325415424 >> 172.21.15.142:0/3325415424 conn(0x7f2fa00fd470 msgr2=0x7f2fa00ff860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:25:27.159 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:27.161+0000 7f2fa64f5640 1 -- 172.21.15.142:0/3325415424 shutdown_connections 2024-09-06T14:25:27.159 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:27.161+0000 7f2fa64f5640 1 -- 172.21.15.142:0/3325415424 wait complete. 2024-09-06T14:25:27.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:26 smithi162 ceph-mon[33787]: pgmap v18: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:25:27.297 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 06 14:25:27 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-iscsi-iscsi-a[59853]: debug there is no tcmu-runner data available 2024-09-06T14:25:28.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:28 smithi162 ceph-mon[33787]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-06T14:25:28.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:28 smithi162 ceph-mon[33787]: from='client.? 172.21.15.142:0/3325415424' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:25:28.459 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:28 smithi142 ceph-mon[29958]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-06T14:25:28.459 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:28 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/3325415424' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:25:28.459 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:28 smithi142 ceph-mon[37372]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-06T14:25:28.459 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:28 smithi142 ceph-mon[37372]: from='client.? 172.21.15.142:0/3325415424' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:25:29.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:29 smithi162 ceph-mon[33787]: pgmap v19: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:25:29.458 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:29 smithi142 ceph-mon[29958]: pgmap v19: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:25:29.459 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:29 smithi142 ceph-mon[37372]: pgmap v19: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:25:31.046 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph health --format=json 2024-09-06T14:25:31.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:31 smithi162 ceph-mon[33787]: pgmap v20: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:25:31.302 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:25:31.423 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:31 smithi142 ceph-mon[29958]: pgmap v20: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:25:31.423 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:31 smithi142 ceph-mon[37372]: pgmap v20: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:25:31.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:31.808+0000 7f8c875a1640 1 -- 172.21.15.142:0/1670383950 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8c80101580 msgr2=0x7f8c80101960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:25:31.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:31.808+0000 7f8c875a1640 1 --2- 172.21.15.142:0/1670383950 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8c80101580 0x7f8c80101960 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f8c78009ca0 tx=0x7f8c7802a8c0 comp rx=0 tx=0).stop 2024-09-06T14:25:31.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:31.808+0000 7f8c875a1640 1 -- 172.21.15.142:0/1670383950 shutdown_connections 2024-09-06T14:25:31.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:31.808+0000 7f8c875a1640 1 --2- 172.21.15.142:0/1670383950 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8c8010f130 0x7f8c80111520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:31.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:31.808+0000 7f8c875a1640 1 --2- 172.21.15.142:0/1670383950 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8c80101ea0 0x7f8c8010ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:31.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:31.808+0000 7f8c875a1640 1 --2- 172.21.15.142:0/1670383950 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8c80101580 0x7f8c80101960 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:31.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:31.808+0000 7f8c875a1640 1 -- 172.21.15.142:0/1670383950 >> 172.21.15.142:0/1670383950 conn(0x7f8c800fd450 msgr2=0x7f8c800ff870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:25:31.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:31.808+0000 7f8c875a1640 1 -- 172.21.15.142:0/1670383950 shutdown_connections 2024-09-06T14:25:31.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:31.808+0000 7f8c875a1640 1 -- 172.21.15.142:0/1670383950 wait complete. 2024-09-06T14:25:31.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:31.809+0000 7f8c875a1640 1 Processor -- start 2024-09-06T14:25:31.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:31.809+0000 7f8c875a1640 1 -- start start 2024-09-06T14:25:31.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:31.809+0000 7f8c875a1640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8c80101580 0x7f8c8019a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:25:31.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:31.809+0000 7f8c875a1640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8c80101ea0 0x7f8c8019aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:25:31.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:31.809+0000 7f8c875a1640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8c8010f130 0x7f8c801a0260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:25:31.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:31.809+0000 7f8c875a1640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f8c801143c0 con 0x7f8c8010f130 2024-09-06T14:25:31.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:31.809+0000 7f8c875a1640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f8c80114240 con 0x7f8c80101ea0 2024-09-06T14:25:31.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:31.809+0000 7f8c875a1640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7f8c80114540 con 0x7f8c80101580 2024-09-06T14:25:31.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:31.809+0000 7f8c8533d640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8c80101580 0x7f8c8019a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:25:31.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:31.809+0000 7f8c85b3e640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8c8010f130 0x7f8c801a0260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:25:31.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:31.809+0000 7f8c8533d640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8c80101580 0x7f8c8019a980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:39156/0 (socket says 172.21.15.142:39156) 2024-09-06T14:25:31.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:31.809+0000 7f8c8533d640 1 -- 172.21.15.142:0/1426405605 learned_addr learned my addr 172.21.15.142:0/1426405605 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:25:31.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:31.809+0000 7f8c84b3c640 1 --2- 172.21.15.142:0/1426405605 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8c80101ea0 0x7f8c8019aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:25:31.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:31.810+0000 7f8c85b3e640 1 -- 172.21.15.142:0/1426405605 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8c80101580 msgr2=0x7f8c8019a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:25:31.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:31.810+0000 7f8c85b3e640 1 --2- 172.21.15.142:0/1426405605 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8c80101580 0x7f8c8019a980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:31.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:31.810+0000 7f8c85b3e640 1 -- 172.21.15.142:0/1426405605 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8c80101ea0 msgr2=0x7f8c8019aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:25:31.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:31.810+0000 7f8c85b3e640 1 --2- 172.21.15.142:0/1426405605 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8c80101ea0 0x7f8c8019aec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:31.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:31.810+0000 7f8c85b3e640 1 -- 172.21.15.142:0/1426405605 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c801a0a30 con 0x7f8c8010f130 2024-09-06T14:25:31.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:31.810+0000 7f8c8533d640 1 --2- 172.21.15.142:0/1426405605 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8c80101580 0x7f8c8019a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-06T14:25:31.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:31.810+0000 7f8c85b3e640 1 --2- 172.21.15.142:0/1426405605 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8c8010f130 0x7f8c801a0260 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f8c7000d650 tx=0x7f8c7000db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:25:31.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:31.810+0000 7f8c7e7fc640 1 -- 172.21.15.142:0/1426405605 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c70017070 con 0x7f8c8010f130 2024-09-06T14:25:31.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:31.810+0000 7f8c875a1640 1 -- 172.21.15.142:0/1426405605 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c801a0d20 con 0x7f8c8010f130 2024-09-06T14:25:31.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:31.810+0000 7f8c7e7fc640 1 -- 172.21.15.142:0/1426405605 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7f8c70009820 con 0x7f8c8010f130 2024-09-06T14:25:31.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:31.810+0000 7f8c7e7fc640 1 -- 172.21.15.142:0/1426405605 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c70004b10 con 0x7f8c8010f130 2024-09-06T14:25:31.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:31.810+0000 7f8c875a1640 1 -- 172.21.15.142:0/1426405605 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c801a73e0 con 0x7f8c8010f130 2024-09-06T14:25:31.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:31.811+0000 7f8c875a1640 1 -- 172.21.15.142:0/1426405605 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c80101960 con 0x7f8c8010f130 2024-09-06T14:25:31.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:31.811+0000 7f8c7e7fc640 1 -- 172.21.15.142:0/1426405605 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 22) ==== 99844+0+0 (secure 0 0 0) 0x7f8c70004cb0 con 0x7f8c8010f130 2024-09-06T14:25:31.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:31.812+0000 7f8c7e7fc640 1 --2- 172.21.15.142:0/1426405605 >> v2:172.21.15.142:6800/2944545612 conn(0x7f8c68077620 0x7f8c68079ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:25:31.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:31.812+0000 7f8c7e7fc640 1 -- 172.21.15.142:0/1426405605 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 6043+0+0 (secure 0 0 0) 0x7f8c70099130 con 0x7f8c8010f130 2024-09-06T14:25:31.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:31.812+0000 7f8c8533d640 1 --2- 172.21.15.142:0/1426405605 >> v2:172.21.15.142:6800/2944545612 conn(0x7f8c68077620 0x7f8c68079ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:25:31.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:31.812+0000 7f8c8533d640 1 --2- 172.21.15.142:0/1426405605 >> v2:172.21.15.142:6800/2944545612 conn(0x7f8c68077620 0x7f8c68079ae0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f8c78005fe0 tx=0x7f8c78005f50 comp rx=0 tx=0).ready entity=mgr.14601 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:25:31.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:31.816+0000 7f8c7e7fc640 1 -- 172.21.15.142:0/1426405605 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7f8c700663c0 con 0x7f8c8010f130 2024-09-06T14:25:32.001 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:32.004+0000 7f8c875a1640 1 -- 172.21.15.142:0/1426405605 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f8c8010a2c0 con 0x7f8c8010f130 2024-09-06T14:25:32.002 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:32.005+0000 7f8c7e7fc640 1 -- 172.21.15.142:0/1426405605 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+212 (secure 0 0 0) 0x7f8c7006ade0 con 0x7f8c8010f130 2024-09-06T14:25:32.002 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:25:32.002 INFO:teuthology.orchestra.run.smithi142.stdout:{"status":"HEALTH_WARN","checks":{"MGR_MODULE_DEPENDENCY":{"severity":"HEALTH_WARN","summary":{"message":"Module 'volumes' has failed dependency: No module named 'ceph.fs'","count":1},"muted":false}},"mutes":[]} 2024-09-06T14:25:32.005 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:32.008+0000 7f8c875a1640 1 -- 172.21.15.142:0/1426405605 >> v2:172.21.15.142:6800/2944545612 conn(0x7f8c68077620 msgr2=0x7f8c68079ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:25:32.005 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:32.008+0000 7f8c875a1640 1 --2- 172.21.15.142:0/1426405605 >> v2:172.21.15.142:6800/2944545612 conn(0x7f8c68077620 0x7f8c68079ae0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f8c78005fe0 tx=0x7f8c78005f50 comp rx=0 tx=0).stop 2024-09-06T14:25:32.005 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:32.008+0000 7f8c875a1640 1 -- 172.21.15.142:0/1426405605 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8c8010f130 msgr2=0x7f8c801a0260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:25:32.005 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:32.008+0000 7f8c875a1640 1 --2- 172.21.15.142:0/1426405605 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8c8010f130 0x7f8c801a0260 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f8c7000d650 tx=0x7f8c7000db10 comp rx=0 tx=0).stop 2024-09-06T14:25:32.005 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:32.008+0000 7f8c875a1640 1 -- 172.21.15.142:0/1426405605 shutdown_connections 2024-09-06T14:25:32.005 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:32.008+0000 7f8c875a1640 1 --2- 172.21.15.142:0/1426405605 >> v2:172.21.15.142:6800/2944545612 conn(0x7f8c68077620 0x7f8c68079ae0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:32.005 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:32.008+0000 7f8c875a1640 1 --2- 172.21.15.142:0/1426405605 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8c8010f130 0x7f8c801a0260 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:32.005 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:32.008+0000 7f8c875a1640 1 --2- 172.21.15.142:0/1426405605 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8c80101ea0 0x7f8c8019aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:32.005 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:32.008+0000 7f8c875a1640 1 --2- 172.21.15.142:0/1426405605 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8c80101580 0x7f8c8019a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:32.006 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:32.008+0000 7f8c875a1640 1 -- 172.21.15.142:0/1426405605 >> 172.21.15.142:0/1426405605 conn(0x7f8c800fd450 msgr2=0x7f8c800ff840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:25:32.006 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:32.008+0000 7f8c875a1640 1 -- 172.21.15.142:0/1426405605 shutdown_connections 2024-09-06T14:25:32.006 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:32.008+0000 7f8c875a1640 1 -- 172.21.15.142:0/1426405605 wait complete. 2024-09-06T14:25:32.387 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:32 smithi162 ceph-mon[33787]: from='client.? 172.21.15.142:0/1426405605' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:25:32.428 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:32 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/1426405605' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:25:32.428 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:32 smithi142 ceph-mon[37372]: from='client.? 172.21.15.142:0/1426405605' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:25:33.349 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:33 smithi162 ceph-mon[33787]: pgmap v21: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:25:33.459 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:33 smithi142 ceph-mon[29958]: pgmap v21: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:25:33.459 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:33 smithi142 ceph-mon[37372]: pgmap v21: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:25:34.209 INFO:journalctl@ceph.alertmanager.a.smithi142.stdout:Sep 06 14:25:33 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-alertmanager-a[70815]: ts=2024-09-06T14:25:33.856Z caller=cluster.go:700 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.002979108s 2024-09-06T14:25:34.709 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:25:34 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: ::ffff:172.21.15.162 - - [06/Sep/2024:14:25:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-06T14:25:35.047 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:34 smithi162 systemd[1]: Starting Ceph grafana.a for ce32462e-6c5a-11ef-bce4-c7b262605968... 2024-09-06T14:25:35.376 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 podman[62757]: 2024-09-06 14:25:35.031712571 +0000 UTC m=+0.016436472 image pull c8b91775d855b99270fc5d22f3c6737e8cca01ef4c25c8b0362295e0746fa39b quay.io/ceph/grafana:10.4.0 2024-09-06T14:25:35.377 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 podman[62757]: 2024-09-06 14:25:35.161187176 +0000 UTC m=+0.145911074 container create bfa2d4b5a0ed274434302ea3bcc86b1f5b5a118227dfc17dfef6cb9c1582d7bb (image=quay.io/ceph/grafana:10.4.0, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a, maintainer=Grafana Labs ) 2024-09-06T14:25:35.377 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-mon[33787]: pgmap v22: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:25:35.459 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:35 smithi142 ceph-mon[29958]: pgmap v22: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:25:35.459 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:35 smithi142 ceph-mon[37372]: pgmap v22: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:25:35.627 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 podman[62757]: 2024-09-06 14:25:35.380211091 +0000 UTC m=+0.364934996 container init bfa2d4b5a0ed274434302ea3bcc86b1f5b5a118227dfc17dfef6cb9c1582d7bb (image=quay.io/ceph/grafana:10.4.0, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a, maintainer=Grafana Labs ) 2024-09-06T14:25:35.627 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 podman[62757]: 2024-09-06 14:25:35.383139272 +0000 UTC m=+0.367863180 container start bfa2d4b5a0ed274434302ea3bcc86b1f5b5a118227dfc17dfef6cb9c1582d7bb (image=quay.io/ceph/grafana:10.4.0, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a, maintainer=Grafana Labs ) 2024-09-06T14:25:35.628 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 bash[62757]: bfa2d4b5a0ed274434302ea3bcc86b1f5b5a118227dfc17dfef6cb9c1582d7bb 2024-09-06T14:25:35.628 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 systemd[1]: Started Ceph grafana.a for ce32462e-6c5a-11ef-bce4-c7b262605968. 2024-09-06T14:25:35.628 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=settings t=2024-09-06T14:25:35.508847026Z level=info msg="Starting Grafana" version=10.4.0 commit=03f502a94d17f7dc4e6c34acdf8428aedd986e4c branch=HEAD compiled=2024-09-06T14:25:35Z 2024-09-06T14:25:35.628 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=settings t=2024-09-06T14:25:35.509102634Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-09-06T14:25:35.628 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=settings t=2024-09-06T14:25:35.509111059Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-09-06T14:25:35.628 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=settings t=2024-09-06T14:25:35.509114334Z level=info msg="Config overridden from command line" arg="default.paths.data=/var/lib/grafana" 2024-09-06T14:25:35.628 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=settings t=2024-09-06T14:25:35.509117311Z level=info msg="Config overridden from command line" arg="default.paths.logs=/var/log/grafana" 2024-09-06T14:25:35.628 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=settings t=2024-09-06T14:25:35.509120093Z level=info msg="Config overridden from command line" arg="default.paths.plugins=/var/lib/grafana/plugins" 2024-09-06T14:25:35.628 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=settings t=2024-09-06T14:25:35.509122852Z level=info msg="Config overridden from command line" arg="default.paths.provisioning=/etc/grafana/provisioning" 2024-09-06T14:25:35.628 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=settings t=2024-09-06T14:25:35.509125523Z level=info msg="Config overridden from command line" arg="default.log.mode=console" 2024-09-06T14:25:35.628 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=settings t=2024-09-06T14:25:35.509128448Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-09-06T14:25:35.629 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=settings t=2024-09-06T14:25:35.509131181Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-09-06T14:25:35.629 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=settings t=2024-09-06T14:25:35.509133849Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-09-06T14:25:35.629 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=settings t=2024-09-06T14:25:35.509136963Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-09-06T14:25:35.629 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=settings t=2024-09-06T14:25:35.509139922Z level=info msg=Target target=[all] 2024-09-06T14:25:35.629 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=settings t=2024-09-06T14:25:35.509146382Z level=info msg="Path Home" path=/usr/share/grafana 2024-09-06T14:25:35.629 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=settings t=2024-09-06T14:25:35.509148949Z level=info msg="Path Data" path=/var/lib/grafana 2024-09-06T14:25:35.629 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=settings t=2024-09-06T14:25:35.50915142Z level=info msg="Path Logs" path=/var/log/grafana 2024-09-06T14:25:35.629 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=settings t=2024-09-06T14:25:35.509153845Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-09-06T14:25:35.629 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=settings t=2024-09-06T14:25:35.509156368Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-09-06T14:25:35.629 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=settings t=2024-09-06T14:25:35.509162372Z level=info msg="App mode production" 2024-09-06T14:25:35.629 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=sqlstore t=2024-09-06T14:25:35.509430486Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-09-06T14:25:35.629 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=sqlstore t=2024-09-06T14:25:35.509446165Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-09-06T14:25:35.630 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.509895495Z level=info msg="Starting DB migrations" 2024-09-06T14:25:35.630 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.511039398Z level=info msg="Executing migration" id="create migration_log table" 2024-09-06T14:25:35.630 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.512150535Z level=info msg="Migration successfully executed" id="create migration_log table" duration=1.110507ms 2024-09-06T14:25:35.630 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.512749557Z level=info msg="Executing migration" id="create user table" 2024-09-06T14:25:35.630 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.513504867Z level=info msg="Migration successfully executed" id="create user table" duration=753.901µs 2024-09-06T14:25:35.630 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.513999422Z level=info msg="Executing migration" id="add unique index user.login" 2024-09-06T14:25:35.630 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.5146205Z level=info msg="Migration successfully executed" id="add unique index user.login" duration=621.22µs 2024-09-06T14:25:35.630 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.514989002Z level=info msg="Executing migration" id="add unique index user.email" 2024-09-06T14:25:35.630 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.515437261Z level=info msg="Migration successfully executed" id="add unique index user.email" duration=448.161µs 2024-09-06T14:25:35.630 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.515857598Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-09-06T14:25:35.630 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.516300376Z level=info msg="Migration successfully executed" id="drop index UQE_user_login - v1" duration=441.042µs 2024-09-06T14:25:35.630 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.516841606Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-09-06T14:25:35.631 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.51729046Z level=info msg="Migration successfully executed" id="drop index UQE_user_email - v1" duration=448.666µs 2024-09-06T14:25:35.632 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.519646882Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-09-06T14:25:35.632 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.521338394Z level=info msg="Migration successfully executed" id="Rename table user to user_v1 - v1" duration=1.690909ms 2024-09-06T14:25:35.632 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.52252306Z level=info msg="Executing migration" id="create user table v2" 2024-09-06T14:25:35.632 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.523115278Z level=info msg="Migration successfully executed" id="create user table v2" duration=592.615µs 2024-09-06T14:25:35.632 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.525410658Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-09-06T14:25:35.632 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.525984842Z level=info msg="Migration successfully executed" id="create index UQE_user_login - v2" duration=575.105µs 2024-09-06T14:25:35.632 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.528206039Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-09-06T14:25:35.632 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.528679143Z level=info msg="Migration successfully executed" id="create index UQE_user_email - v2" duration=473.049µs 2024-09-06T14:25:35.632 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.530989096Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-09-06T14:25:35.632 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.531300812Z level=info msg="Migration successfully executed" id="copy data_source v1 to v2" duration=311.709µs 2024-09-06T14:25:35.633 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.53370432Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-09-06T14:25:35.633 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.53411298Z level=info msg="Migration successfully executed" id="Drop old table user_v1" duration=408.728µs 2024-09-06T14:25:35.633 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.536425068Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-09-06T14:25:35.633 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.537230418Z level=info msg="Migration successfully executed" id="Add column help_flags1 to user table" duration=804.762µs 2024-09-06T14:25:35.633 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.537609388Z level=info msg="Executing migration" id="Update user table charset" 2024-09-06T14:25:35.633 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.537627565Z level=info msg="Migration successfully executed" id="Update user table charset" duration=18.919µs 2024-09-06T14:25:35.633 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.540479465Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-09-06T14:25:35.633 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.541306204Z level=info msg="Migration successfully executed" id="Add last_seen_at column to user" duration=826.711µs 2024-09-06T14:25:35.633 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.543310689Z level=info msg="Executing migration" id="Add missing user data" 2024-09-06T14:25:35.633 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.543464086Z level=info msg="Migration successfully executed" id="Add missing user data" duration=153.019µs 2024-09-06T14:25:35.633 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.54600748Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-09-06T14:25:35.633 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.546742236Z level=info msg="Migration successfully executed" id="Add is_disabled column to user" duration=734.37µs 2024-09-06T14:25:35.633 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.547087009Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-09-06T14:25:35.634 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.547548696Z level=info msg="Migration successfully executed" id="Add index user.login/user.email" duration=459.859µs 2024-09-06T14:25:35.634 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.549924026Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-09-06T14:25:35.634 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.550687242Z level=info msg="Migration successfully executed" id="Add is_service_account column to user" duration=762.763µs 2024-09-06T14:25:35.634 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.551045963Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-09-06T14:25:35.634 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.556241418Z level=info msg="Migration successfully executed" id="Update is_service_account column to nullable" duration=5.193302ms 2024-09-06T14:25:35.634 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.556632289Z level=info msg="Executing migration" id="Add uid column to user" 2024-09-06T14:25:35.634 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.557368927Z level=info msg="Migration successfully executed" id="Add uid column to user" duration=737.249µs 2024-09-06T14:25:35.634 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.559521361Z level=info msg="Executing migration" id="Update uid column values for users" 2024-09-06T14:25:35.634 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.559654012Z level=info msg="Migration successfully executed" id="Update uid column values for users" duration=132.628µs 2024-09-06T14:25:35.634 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.562339629Z level=info msg="Executing migration" id="Add unique index user_uid" 2024-09-06T14:25:35.634 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.562789013Z level=info msg="Migration successfully executed" id="Add unique index user_uid" duration=449.099µs 2024-09-06T14:25:35.634 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.565380884Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-09-06T14:25:35.634 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.565897703Z level=info msg="Migration successfully executed" id="create temp user table v1-7" duration=516.558µs 2024-09-06T14:25:35.635 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.568139127Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-09-06T14:25:35.635 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.568692781Z level=info msg="Migration successfully executed" id="create index IDX_temp_user_email - v1-7" duration=555.947µs 2024-09-06T14:25:35.635 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.570667395Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-09-06T14:25:35.635 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.571116468Z level=info msg="Migration successfully executed" id="create index IDX_temp_user_org_id - v1-7" duration=449.453µs 2024-09-06T14:25:35.635 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.573195874Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-09-06T14:25:35.635 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.573643133Z level=info msg="Migration successfully executed" id="create index IDX_temp_user_code - v1-7" duration=446.901µs 2024-09-06T14:25:35.635 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.574008989Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-09-06T14:25:35.635 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.574448525Z level=info msg="Migration successfully executed" id="create index IDX_temp_user_status - v1-7" duration=438.681µs 2024-09-06T14:25:35.635 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.576654299Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-09-06T14:25:35.635 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.576672151Z level=info msg="Migration successfully executed" id="Update temp_user table charset" duration=18.467µs 2024-09-06T14:25:35.635 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.579239806Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-09-06T14:25:35.635 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.579704262Z level=info msg="Migration successfully executed" id="drop index IDX_temp_user_email - v1" duration=464.243µs 2024-09-06T14:25:35.635 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.582074361Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-09-06T14:25:35.635 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.58259984Z level=info msg="Migration successfully executed" id="drop index IDX_temp_user_org_id - v1" duration=527.617µs 2024-09-06T14:25:35.636 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.584580008Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-09-06T14:25:35.636 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.585030754Z level=info msg="Migration successfully executed" id="drop index IDX_temp_user_code - v1" duration=450.583µs 2024-09-06T14:25:35.636 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.587254815Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-09-06T14:25:35.636 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.587699754Z level=info msg="Migration successfully executed" id="drop index IDX_temp_user_status - v1" duration=444.779µs 2024-09-06T14:25:35.636 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.589633898Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-09-06T14:25:35.636 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.591604485Z level=info msg="Migration successfully executed" id="Rename table temp_user to temp_user_tmp_qwerty - v1" duration=1.970148ms 2024-09-06T14:25:35.636 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.592268184Z level=info msg="Executing migration" id="create temp_user v2" 2024-09-06T14:25:35.636 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.592818593Z level=info msg="Migration successfully executed" id="create temp_user v2" duration=549.241µs 2024-09-06T14:25:35.636 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.594995958Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-09-06T14:25:35.636 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.595461868Z level=info msg="Migration successfully executed" id="create index IDX_temp_user_email - v2" duration=465.684µs 2024-09-06T14:25:35.636 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.597683271Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-09-06T14:25:35.636 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.598143267Z level=info msg="Migration successfully executed" id="create index IDX_temp_user_org_id - v2" duration=459.823µs 2024-09-06T14:25:35.636 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.600268379Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-09-06T14:25:35.636 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.600878273Z level=info msg="Migration successfully executed" id="create index IDX_temp_user_code - v2" duration=611.622µs 2024-09-06T14:25:35.636 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.60272926Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-09-06T14:25:35.637 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.603169364Z level=info msg="Migration successfully executed" id="create index IDX_temp_user_status - v2" duration=439.724µs 2024-09-06T14:25:35.637 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.60533672Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-09-06T14:25:35.637 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.605593925Z level=info msg="Migration successfully executed" id="copy temp_user v1 to v2" duration=256.134µs 2024-09-06T14:25:35.637 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.608096721Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-09-06T14:25:35.637 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.608497782Z level=info msg="Migration successfully executed" id="drop temp_user_tmp_qwerty" duration=404.356µs 2024-09-06T14:25:35.637 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.608838445Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-09-06T14:25:35.637 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.609072693Z level=info msg="Migration successfully executed" id="Set created for temp users that will otherwise prematurely expire" duration=233.958µs 2024-09-06T14:25:35.637 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.609668889Z level=info msg="Executing migration" id="create star table" 2024-09-06T14:25:35.637 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.610079949Z level=info msg="Migration successfully executed" id="create star table" duration=410.794µs 2024-09-06T14:25:35.637 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.610534265Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-09-06T14:25:35.637 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.611003282Z level=info msg="Migration successfully executed" id="add unique index star.user_id_dashboard_id" duration=468.732µs 2024-09-06T14:25:35.638 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.611416425Z level=info msg="Executing migration" id="create org table v1" 2024-09-06T14:25:35.638 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.611955975Z level=info msg="Migration successfully executed" id="create org table v1" duration=541.281µs 2024-09-06T14:25:35.638 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.612315149Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-09-06T14:25:35.638 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.612799741Z level=info msg="Migration successfully executed" id="create index UQE_org_name - v1" duration=484.273µs 2024-09-06T14:25:35.638 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.613151835Z level=info msg="Executing migration" id="create org_user table v1" 2024-09-06T14:25:35.638 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.613612131Z level=info msg="Migration successfully executed" id="create org_user table v1" duration=460.024µs 2024-09-06T14:25:35.638 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.613969578Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-09-06T14:25:35.638 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.614450735Z level=info msg="Migration successfully executed" id="create index IDX_org_user_org_id - v1" duration=480.884µs 2024-09-06T14:25:35.638 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.614824522Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-09-06T14:25:35.638 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.61530119Z level=info msg="Migration successfully executed" id="create index UQE_org_user_org_id_user_id - v1" duration=475.762µs 2024-09-06T14:25:35.638 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.615656536Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-09-06T14:25:35.638 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.616118289Z level=info msg="Migration successfully executed" id="create index IDX_org_user_user_id - v1" duration=461.278µs 2024-09-06T14:25:35.638 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.616480451Z level=info msg="Executing migration" id="Update org table charset" 2024-09-06T14:25:35.639 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.616496979Z level=info msg="Migration successfully executed" id="Update org table charset" duration=16.802µs 2024-09-06T14:25:35.639 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.617484025Z level=info msg="Executing migration" id="Update org_user table charset" 2024-09-06T14:25:35.639 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.617535262Z level=info msg="Migration successfully executed" id="Update org_user table charset" duration=73.446µs 2024-09-06T14:25:35.639 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.61817353Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-09-06T14:25:35.639 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.618292544Z level=info msg="Migration successfully executed" id="Migrate all Read Only Viewers to Viewers" duration=118.971µs 2024-09-06T14:25:35.639 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.619226659Z level=info msg="Executing migration" id="create dashboard table" 2024-09-06T14:25:35.639 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.619887505Z level=info msg="Migration successfully executed" id="create dashboard table" duration=663.064µs 2024-09-06T14:25:35.639 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.62024355Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-09-06T14:25:35.639 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.620761063Z level=info msg="Migration successfully executed" id="add index dashboard.account_id" duration=516.983µs 2024-09-06T14:25:35.639 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.621121752Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-09-06T14:25:35.639 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.621626331Z level=info msg="Migration successfully executed" id="add unique index dashboard_account_id_slug" duration=504.143µs 2024-09-06T14:25:35.639 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.621967579Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-09-06T14:25:35.639 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.62238224Z level=info msg="Migration successfully executed" id="create dashboard_tag table" duration=414.408µs 2024-09-06T14:25:35.639 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.622807092Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-09-06T14:25:35.639 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.62330438Z level=info msg="Migration successfully executed" id="add unique index dashboard_tag.dasboard_id_term" duration=496.002µs 2024-09-06T14:25:35.640 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.623678684Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-09-06T14:25:35.640 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.624172227Z level=info msg="Migration successfully executed" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" duration=493.479µs 2024-09-06T14:25:35.640 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.624535581Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-09-06T14:25:35.640 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.627980243Z level=info msg="Migration successfully executed" id="Rename table dashboard to dashboard_v1 - v1" duration=3.444254ms 2024-09-06T14:25:35.640 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.628353205Z level=info msg="Executing migration" id="create dashboard v2" 2024-09-06T14:25:35.640 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.628869979Z level=info msg="Migration successfully executed" id="create dashboard v2" duration=516.625µs 2024-09-06T14:25:35.640 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.629223718Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-09-06T14:25:35.640 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.629695132Z level=info msg="Migration successfully executed" id="create index IDX_dashboard_org_id - v2" duration=471.078µs 2024-09-06T14:25:35.640 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.630091354Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-09-06T14:25:35.640 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.630592811Z level=info msg="Migration successfully executed" id="create index UQE_dashboard_org_id_slug - v2" duration=501.111µs 2024-09-06T14:25:35.640 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.630993496Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-09-06T14:25:35.640 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.631271262Z level=info msg="Migration successfully executed" id="copy dashboard v1 to v2" duration=278.013µs 2024-09-06T14:25:35.640 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.631829121Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-09-06T14:25:35.641 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.632442041Z level=info msg="Migration successfully executed" id="drop table dashboard_v1" duration=612.562µs 2024-09-06T14:25:35.641 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.632819154Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-09-06T14:25:35.641 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.632869535Z level=info msg="Migration successfully executed" id="alter dashboard.data to mediumtext v1" duration=50.597µs 2024-09-06T14:25:35.641 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.633654955Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-09-06T14:25:35.641 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.634780212Z level=info msg="Migration successfully executed" id="Add column updated_by in dashboard - v2" duration=1.12508ms 2024-09-06T14:25:35.740 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph health --format=json 2024-09-06T14:25:35.879 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.635146706Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-09-06T14:25:35.879 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.63624532Z level=info msg="Migration successfully executed" id="Add column created_by in dashboard - v2" duration=1.09815ms 2024-09-06T14:25:35.879 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.636602844Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-09-06T14:25:35.879 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.637698473Z level=info msg="Migration successfully executed" id="Add column gnetId in dashboard" duration=1.095199ms 2024-09-06T14:25:35.879 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.638065267Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-09-06T14:25:35.879 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.638609353Z level=info msg="Migration successfully executed" id="Add index for gnetId in dashboard" duration=543.838µs 2024-09-06T14:25:35.879 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.638976779Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-09-06T14:25:35.879 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.640094038Z level=info msg="Migration successfully executed" id="Add column plugin_id in dashboard" duration=1.118082ms 2024-09-06T14:25:35.879 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.640454557Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-09-06T14:25:35.879 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.640962197Z level=info msg="Migration successfully executed" id="Add index for plugin_id in dashboard" duration=507.212µs 2024-09-06T14:25:35.879 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.64132861Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-09-06T14:25:35.880 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.641812345Z level=info msg="Migration successfully executed" id="Add index for dashboard_id in dashboard_tag" duration=483.45µs 2024-09-06T14:25:35.880 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.642175853Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-09-06T14:25:35.880 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.642194723Z level=info msg="Migration successfully executed" id="Update dashboard table charset" duration=18.924µs 2024-09-06T14:25:35.880 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.643043562Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-09-06T14:25:35.880 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.643062537Z level=info msg="Migration successfully executed" id="Update dashboard_tag table charset" duration=19.148µs 2024-09-06T14:25:35.880 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.643936751Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-09-06T14:25:35.880 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.645117687Z level=info msg="Migration successfully executed" id="Add column folder_id in dashboard" duration=1.180551ms 2024-09-06T14:25:35.880 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.646625514Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-09-06T14:25:35.880 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.647806082Z level=info msg="Migration successfully executed" id="Add column isFolder in dashboard" duration=1.180121ms 2024-09-06T14:25:35.880 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.64953824Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-09-06T14:25:35.880 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.650717473Z level=info msg="Migration successfully executed" id="Add column has_acl in dashboard" duration=1.178889ms 2024-09-06T14:25:35.881 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.652418045Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-09-06T14:25:35.881 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.65357861Z level=info msg="Migration successfully executed" id="Add column uid in dashboard" duration=1.159208ms 2024-09-06T14:25:35.881 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.655490112Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-09-06T14:25:35.881 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.65565855Z level=info msg="Migration successfully executed" id="Update uid column values in dashboard" duration=171.128µs 2024-09-06T14:25:35.881 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.658247094Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-09-06T14:25:35.881 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.65875563Z level=info msg="Migration successfully executed" id="Add unique index dashboard_org_id_uid" duration=509.325µs 2024-09-06T14:25:35.881 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.659120707Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-09-06T14:25:35.881 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.65959336Z level=info msg="Migration successfully executed" id="Remove unique index org_id_slug" duration=472.408µs 2024-09-06T14:25:35.881 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.659976198Z level=info msg="Executing migration" id="Update dashboard title length" 2024-09-06T14:25:35.881 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.659996988Z level=info msg="Migration successfully executed" id="Update dashboard title length" duration=21.013µs 2024-09-06T14:25:35.881 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.660970559Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-09-06T14:25:35.881 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.661567524Z level=info msg="Migration successfully executed" id="Add unique index for dashboard_org_id_title_folder_id" duration=598.847µs 2024-09-06T14:25:35.881 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.661930186Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-09-06T14:25:35.883 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.662388592Z level=info msg="Migration successfully executed" id="create dashboard_provisioning" duration=458.144µs 2024-09-06T14:25:35.883 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.662781087Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-09-06T14:25:35.883 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.665849653Z level=info msg="Migration successfully executed" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" duration=3.06845ms 2024-09-06T14:25:35.883 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.666211115Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-09-06T14:25:35.884 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.666806771Z level=info msg="Migration successfully executed" id="create dashboard_provisioning v2" duration=595.12µs 2024-09-06T14:25:35.884 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.667173056Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-09-06T14:25:35.884 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.667680483Z level=info msg="Migration successfully executed" id="create index IDX_dashboard_provisioning_dashboard_id - v2" duration=506µs 2024-09-06T14:25:35.884 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.668065657Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-09-06T14:25:35.884 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.668559071Z level=info msg="Migration successfully executed" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" duration=493.253µs 2024-09-06T14:25:35.884 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.668933631Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-09-06T14:25:35.884 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.669152676Z level=info msg="Migration successfully executed" id="copy dashboard_provisioning v1 to v2" duration=219.589µs 2024-09-06T14:25:35.884 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.669938123Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-09-06T14:25:35.884 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.670474575Z level=info msg="Migration successfully executed" id="drop dashboard_provisioning_tmp_qwerty" duration=538.001µs 2024-09-06T14:25:35.884 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.670820223Z level=info msg="Executing migration" id="Add check_sum column" 2024-09-06T14:25:35.884 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.672136872Z level=info msg="Migration successfully executed" id="Add check_sum column" duration=1.316457ms 2024-09-06T14:25:35.884 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.672522265Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-09-06T14:25:35.885 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.672992206Z level=info msg="Migration successfully executed" id="Add index for dashboard_title" duration=469.594µs 2024-09-06T14:25:35.885 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.67336436Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-09-06T14:25:35.885 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.67348712Z level=info msg="Migration successfully executed" id="delete tags for deleted dashboards" duration=122.69µs 2024-09-06T14:25:35.888 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.674247759Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-09-06T14:25:35.888 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.67437881Z level=info msg="Migration successfully executed" id="delete stars for deleted dashboards" duration=132.007µs 2024-09-06T14:25:35.888 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.675059817Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-09-06T14:25:35.888 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.675570099Z level=info msg="Migration successfully executed" id="Add index for dashboard_is_folder" duration=510.111µs 2024-09-06T14:25:35.888 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.675925461Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-09-06T14:25:35.888 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.677206582Z level=info msg="Migration successfully executed" id="Add isPublic for dashboard" duration=1.280416ms 2024-09-06T14:25:35.888 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.677607422Z level=info msg="Executing migration" id="create data_source table" 2024-09-06T14:25:35.888 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.678216399Z level=info msg="Migration successfully executed" id="create data_source table" duration=607.034µs 2024-09-06T14:25:35.888 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.678610023Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-09-06T14:25:35.888 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.679115529Z level=info msg="Migration successfully executed" id="add index data_source.account_id" duration=505.575µs 2024-09-06T14:25:35.889 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.679480784Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-09-06T14:25:35.889 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.679967736Z level=info msg="Migration successfully executed" id="add unique index data_source.account_id_name" duration=487.294µs 2024-09-06T14:25:35.889 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.680322888Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-09-06T14:25:35.889 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.68079376Z level=info msg="Migration successfully executed" id="drop index IDX_data_source_account_id - v1" duration=470.783µs 2024-09-06T14:25:35.889 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.681212994Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-09-06T14:25:35.889 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.681692439Z level=info msg="Migration successfully executed" id="drop index UQE_data_source_account_id_name - v1" duration=478.346µs 2024-09-06T14:25:35.889 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.682062544Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-09-06T14:25:35.889 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.685389004Z level=info msg="Migration successfully executed" id="Rename table data_source to data_source_v1 - v1" duration=3.326222ms 2024-09-06T14:25:35.889 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.685784378Z level=info msg="Executing migration" id="create data_source table v2" 2024-09-06T14:25:35.889 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.686372668Z level=info msg="Migration successfully executed" id="create data_source table v2" duration=587.977µs 2024-09-06T14:25:35.889 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.68675957Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-09-06T14:25:35.889 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.687298668Z level=info msg="Migration successfully executed" id="create index IDX_data_source_org_id - v2" duration=540.034µs 2024-09-06T14:25:35.890 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.687675Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-09-06T14:25:35.890 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.688199508Z level=info msg="Migration successfully executed" id="create index UQE_data_source_org_id_name - v2" duration=524.121µs 2024-09-06T14:25:35.890 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.690629081Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-09-06T14:25:35.890 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.691047326Z level=info msg="Migration successfully executed" id="Drop old table data_source_v1 #2" duration=420.618µs 2024-09-06T14:25:35.890 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.693224219Z level=info msg="Executing migration" id="Add column with_credentials" 2024-09-06T14:25:35.890 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.694650074Z level=info msg="Migration successfully executed" id="Add column with_credentials" duration=1.425451ms 2024-09-06T14:25:35.890 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.696209828Z level=info msg="Executing migration" id="Add secure json data column" 2024-09-06T14:25:35.890 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.697588137Z level=info msg="Migration successfully executed" id="Add secure json data column" duration=1.378053ms 2024-09-06T14:25:35.890 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.698917919Z level=info msg="Executing migration" id="Update data_source table charset" 2024-09-06T14:25:35.890 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.698954214Z level=info msg="Migration successfully executed" id="Update data_source table charset" duration=36.434µs 2024-09-06T14:25:35.890 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.701740717Z level=info msg="Executing migration" id="Update initial version to 1" 2024-09-06T14:25:35.890 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.70187859Z level=info msg="Migration successfully executed" id="Update initial version to 1" duration=136.921µs 2024-09-06T14:25:35.890 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.702281248Z level=info msg="Executing migration" id="Add read_only data column" 2024-09-06T14:25:35.890 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.703684604Z level=info msg="Migration successfully executed" id="Add read_only data column" duration=1.403846ms 2024-09-06T14:25:35.890 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.704066949Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-09-06T14:25:35.891 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.704205955Z level=info msg="Migration successfully executed" id="Migrate logging ds to loki ds" duration=137.153µs 2024-09-06T14:25:35.891 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.706871149Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-09-06T14:25:35.891 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.707013034Z level=info msg="Migration successfully executed" id="Update json_data with nulls" duration=141.677µs 2024-09-06T14:25:35.891 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.707410798Z level=info msg="Executing migration" id="Add uid column" 2024-09-06T14:25:35.891 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.708822801Z level=info msg="Migration successfully executed" id="Add uid column" duration=1.411624ms 2024-09-06T14:25:35.891 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.709220745Z level=info msg="Executing migration" id="Update uid value" 2024-09-06T14:25:35.891 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.709359599Z level=info msg="Migration successfully executed" id="Update uid value" duration=138.706µs 2024-09-06T14:25:35.891 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.709767262Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-09-06T14:25:35.891 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.710294901Z level=info msg="Migration successfully executed" id="Add unique index datasource_org_id_uid" duration=528.513µs 2024-09-06T14:25:35.891 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.712220249Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-09-06T14:25:35.891 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.712757486Z level=info msg="Migration successfully executed" id="add unique index datasource_org_id_is_default" duration=536.996µs 2024-09-06T14:25:35.891 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.714837136Z level=info msg="Executing migration" id="create api_key table" 2024-09-06T14:25:35.891 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.715387864Z level=info msg="Migration successfully executed" id="create api_key table" duration=550.452µs 2024-09-06T14:25:35.891 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.717692779Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-09-06T14:25:35.892 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.718222153Z level=info msg="Migration successfully executed" id="add index api_key.account_id" duration=529.189µs 2024-09-06T14:25:35.892 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.720400262Z level=info msg="Executing migration" id="add index api_key.key" 2024-09-06T14:25:35.892 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.720929765Z level=info msg="Migration successfully executed" id="add index api_key.key" duration=529.327µs 2024-09-06T14:25:35.892 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.723253333Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-09-06T14:25:35.892 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.723820763Z level=info msg="Migration successfully executed" id="add index api_key.account_id_name" duration=566.986µs 2024-09-06T14:25:35.892 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.725928487Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-09-06T14:25:35.892 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.726413062Z level=info msg="Migration successfully executed" id="drop index IDX_api_key_account_id - v1" duration=484.128µs 2024-09-06T14:25:35.892 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.726761103Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-09-06T14:25:35.892 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.727246153Z level=info msg="Migration successfully executed" id="drop index UQE_api_key_key - v1" duration=484.921µs 2024-09-06T14:25:35.892 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.727581114Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-09-06T14:25:35.892 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.728063575Z level=info msg="Migration successfully executed" id="drop index UQE_api_key_account_id_name - v1" duration=482.335µs 2024-09-06T14:25:35.892 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.728406165Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-09-06T14:25:35.892 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.73210443Z level=info msg="Migration successfully executed" id="Rename table api_key to api_key_v1 - v1" duration=3.699111ms 2024-09-06T14:25:35.893 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.732477327Z level=info msg="Executing migration" id="create api_key table v2" 2024-09-06T14:25:35.893 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.732977559Z level=info msg="Migration successfully executed" id="create api_key table v2" duration=500.958µs 2024-09-06T14:25:35.893 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.733342527Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-09-06T14:25:35.893 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.733877618Z level=info msg="Migration successfully executed" id="create index IDX_api_key_org_id - v2" duration=534.02µs 2024-09-06T14:25:35.893 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.73423391Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-09-06T14:25:35.893 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.734744419Z level=info msg="Migration successfully executed" id="create index UQE_api_key_key - v2" duration=510.431µs 2024-09-06T14:25:35.893 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.73509839Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-09-06T14:25:35.893 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.73559399Z level=info msg="Migration successfully executed" id="create index UQE_api_key_org_id_name - v2" duration=495.449µs 2024-09-06T14:25:35.893 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.735977871Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-09-06T14:25:35.893 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.736225393Z level=info msg="Migration successfully executed" id="copy api_key v1 to v2" duration=247.375µs 2024-09-06T14:25:35.893 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.736587525Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-09-06T14:25:35.893 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.736982689Z level=info msg="Migration successfully executed" id="Drop old table api_key_v1" duration=395.15µs 2024-09-06T14:25:35.893 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.737334854Z level=info msg="Executing migration" id="Update api_key table charset" 2024-09-06T14:25:35.893 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.737353133Z level=info msg="Migration successfully executed" id="Update api_key table charset" duration=18.502µs 2024-09-06T14:25:35.894 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.737777588Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-09-06T14:25:35.894 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.739200382Z level=info msg="Migration successfully executed" id="Add expires to api_key table" duration=1.422422ms 2024-09-06T14:25:35.894 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.739562255Z level=info msg="Executing migration" id="Add service account foreign key" 2024-09-06T14:25:35.894 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.741623466Z level=info msg="Migration successfully executed" id="Add service account foreign key" duration=2.060818ms 2024-09-06T14:25:35.894 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.742016189Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-09-06T14:25:35.894 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.742128656Z level=info msg="Migration successfully executed" id="set service account foreign key to nil if 0" duration=112.592µs 2024-09-06T14:25:35.894 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.742531048Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-09-06T14:25:35.894 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.744008897Z level=info msg="Migration successfully executed" id="Add last_used_at to api_key table" duration=1.477499ms 2024-09-06T14:25:35.894 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.744374734Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-09-06T14:25:35.894 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.745923027Z level=info msg="Migration successfully executed" id="Add is_revoked column to api_key table" duration=1.546981ms 2024-09-06T14:25:35.894 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.746292522Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-09-06T14:25:35.894 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.746819865Z level=info msg="Migration successfully executed" id="create dashboard_snapshot table v4" duration=527.03µs 2024-09-06T14:25:35.894 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.747178051Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-09-06T14:25:35.894 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.74756621Z level=info msg="Migration successfully executed" id="drop table dashboard_snapshot_v4 #1" duration=387.888µs 2024-09-06T14:25:35.895 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.747915877Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-09-06T14:25:35.895 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.748486111Z level=info msg="Migration successfully executed" id="create dashboard_snapshot table v5 #2" duration=569.847µs 2024-09-06T14:25:35.895 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.748831027Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-09-06T14:25:35.895 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.749354995Z level=info msg="Migration successfully executed" id="create index UQE_dashboard_snapshot_key - v5" duration=523.782µs 2024-09-06T14:25:35.895 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.749724411Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-09-06T14:25:35.895 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.750264629Z level=info msg="Migration successfully executed" id="create index UQE_dashboard_snapshot_delete_key - v5" duration=539.906µs 2024-09-06T14:25:35.895 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.750608785Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-09-06T14:25:35.895 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.751121018Z level=info msg="Migration successfully executed" id="create index IDX_dashboard_snapshot_user_id - v5" duration=511.933µs 2024-09-06T14:25:35.895 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.751483691Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-09-06T14:25:35.895 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.751532976Z level=info msg="Migration successfully executed" id="alter dashboard_snapshot to mediumtext v2" duration=67.844µs 2024-09-06T14:25:35.895 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.75194797Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-09-06T14:25:35.895 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.751966487Z level=info msg="Migration successfully executed" id="Update dashboard_snapshot table charset" duration=18.464µs 2024-09-06T14:25:35.895 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.75234862Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-09-06T14:25:35.895 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.753995999Z level=info msg="Migration successfully executed" id="Add column external_delete_url to dashboard_snapshots table" duration=1.646906ms 2024-09-06T14:25:35.896 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.75434082Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-09-06T14:25:35.896 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.755976578Z level=info msg="Migration successfully executed" id="Add encrypted dashboard json column" duration=1.635453ms 2024-09-06T14:25:35.896 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.756335067Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-09-06T14:25:35.896 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.756383273Z level=info msg="Migration successfully executed" id="Change dashboard_encrypted column to MEDIUMBLOB" duration=49.5µs 2024-09-06T14:25:35.896 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.756789914Z level=info msg="Executing migration" id="create quota table v1" 2024-09-06T14:25:35.896 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.757252631Z level=info msg="Migration successfully executed" id="create quota table v1" duration=461.167µs 2024-09-06T14:25:35.896 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.757598702Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-09-06T14:25:35.896 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.758114476Z level=info msg="Migration successfully executed" id="create index UQE_quota_org_id_user_id_target - v1" duration=515.472µs 2024-09-06T14:25:35.896 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.75846482Z level=info msg="Executing migration" id="Update quota table charset" 2024-09-06T14:25:35.896 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.758492604Z level=info msg="Migration successfully executed" id="Update quota table charset" duration=27.932µs 2024-09-06T14:25:35.896 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.758887697Z level=info msg="Executing migration" id="create plugin_setting table" 2024-09-06T14:25:35.896 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.75942208Z level=info msg="Migration successfully executed" id="create plugin_setting table" duration=534.084µs 2024-09-06T14:25:35.896 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.759792538Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-09-06T14:25:35.896 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.760356334Z level=info msg="Migration successfully executed" id="create index UQE_plugin_setting_org_id_plugin_id - v1" duration=563.582µs 2024-09-06T14:25:35.896 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.760746401Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-09-06T14:25:35.897 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.762406864Z level=info msg="Migration successfully executed" id="Add column plugin_version to plugin_settings" duration=1.660122ms 2024-09-06T14:25:35.897 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.762772812Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-09-06T14:25:35.897 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.762791912Z level=info msg="Migration successfully executed" id="Update plugin_setting table charset" duration=19.264µs 2024-09-06T14:25:35.897 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.763204269Z level=info msg="Executing migration" id="create session table" 2024-09-06T14:25:35.897 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.763789044Z level=info msg="Migration successfully executed" id="create session table" duration=584.642µs 2024-09-06T14:25:35.897 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.76416749Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-09-06T14:25:35.897 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.764237976Z level=info msg="Migration successfully executed" id="Drop old table playlist table" duration=71.667µs 2024-09-06T14:25:35.897 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.766849827Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-09-06T14:25:35.897 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.76693608Z level=info msg="Migration successfully executed" id="Drop old table playlist_item table" duration=86.388µs 2024-09-06T14:25:35.897 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.767336057Z level=info msg="Executing migration" id="create playlist table v2" 2024-09-06T14:25:35.897 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.767829901Z level=info msg="Migration successfully executed" id="create playlist table v2" duration=493.625µs 2024-09-06T14:25:35.897 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.76821236Z level=info msg="Executing migration" id="create playlist item table v2" 2024-09-06T14:25:35.897 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.768706145Z level=info msg="Migration successfully executed" id="create playlist item table v2" duration=494.134µs 2024-09-06T14:25:35.898 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.769061998Z level=info msg="Executing migration" id="Update playlist table charset" 2024-09-06T14:25:35.898 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.769080467Z level=info msg="Migration successfully executed" id="Update playlist table charset" duration=18.499µs 2024-09-06T14:25:35.898 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.769531777Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-09-06T14:25:35.898 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.769553375Z level=info msg="Migration successfully executed" id="Update playlist_item table charset" duration=18.852µs 2024-09-06T14:25:35.898 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.769972451Z level=info msg="Executing migration" id="Add playlist column created_at" 2024-09-06T14:25:35.898 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.771813905Z level=info msg="Migration successfully executed" id="Add playlist column created_at" duration=1.840988ms 2024-09-06T14:25:35.898 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.772176601Z level=info msg="Executing migration" id="Add playlist column updated_at" 2024-09-06T14:25:35.898 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.773943445Z level=info msg="Migration successfully executed" id="Add playlist column updated_at" duration=1.766431ms 2024-09-06T14:25:35.898 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.7743064Z level=info msg="Executing migration" id="drop preferences table v2" 2024-09-06T14:25:35.898 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.774376003Z level=info msg="Migration successfully executed" id="drop preferences table v2" duration=69.822µs 2024-09-06T14:25:35.898 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.774761971Z level=info msg="Executing migration" id="drop preferences table v3" 2024-09-06T14:25:35.898 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.774830629Z level=info msg="Migration successfully executed" id="drop preferences table v3" duration=68.698µs 2024-09-06T14:25:35.898 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.775212413Z level=info msg="Executing migration" id="create preferences table v3" 2024-09-06T14:25:35.899 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.775742226Z level=info msg="Migration successfully executed" id="create preferences table v3" duration=529.599µs 2024-09-06T14:25:35.899 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.776105525Z level=info msg="Executing migration" id="Update preferences table charset" 2024-09-06T14:25:35.899 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.776124096Z level=info msg="Migration successfully executed" id="Update preferences table charset" duration=18.771µs 2024-09-06T14:25:35.899 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.776539283Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-09-06T14:25:35.899 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.778342513Z level=info msg="Migration successfully executed" id="Add column team_id in preferences" duration=1.802874ms 2024-09-06T14:25:35.899 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.778716976Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-09-06T14:25:35.899 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.778823862Z level=info msg="Migration successfully executed" id="Update team_id column values in preferences" duration=106.953µs 2024-09-06T14:25:35.899 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.779218827Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-09-06T14:25:35.899 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.781063018Z level=info msg="Migration successfully executed" id="Add column week_start in preferences" duration=1.843888ms 2024-09-06T14:25:35.899 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.781433867Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-09-06T14:25:35.899 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.783206491Z level=info msg="Migration successfully executed" id="Add column preferences.json_data" duration=1.773486ms 2024-09-06T14:25:35.899 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.783592413Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-09-06T14:25:35.899 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.783641469Z level=info msg="Migration successfully executed" id="alter preferences.json_data to mediumtext v1" duration=49.356µs 2024-09-06T14:25:35.899 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.784114746Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-09-06T14:25:35.900 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.78472939Z level=info msg="Migration successfully executed" id="Add preferences index org_id" duration=613.689µs 2024-09-06T14:25:35.900 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.785118171Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-09-06T14:25:35.900 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.785663635Z level=info msg="Migration successfully executed" id="Add preferences index user_id" duration=544.908µs 2024-09-06T14:25:35.900 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.786000141Z level=info msg="Executing migration" id="create alert table v1" 2024-09-06T14:25:35.900 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.786748416Z level=info msg="Migration successfully executed" id="create alert table v1" duration=748.062µs 2024-09-06T14:25:35.900 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.788034976Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-09-06T14:25:35.900 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.788754427Z level=info msg="Migration successfully executed" id="add index alert org_id & id " duration=721.404µs 2024-09-06T14:25:35.900 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.790771195Z level=info msg="Executing migration" id="add index alert state" 2024-09-06T14:25:35.900 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.791274527Z level=info msg="Migration successfully executed" id="add index alert state" duration=503.019µs 2024-09-06T14:25:35.900 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.793594224Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-09-06T14:25:35.900 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.794127018Z level=info msg="Migration successfully executed" id="add index alert dashboard_id" duration=532.543µs 2024-09-06T14:25:35.900 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.796657894Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-09-06T14:25:35.900 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.797196539Z level=info msg="Migration successfully executed" id="Create alert_rule_tag table v1" duration=542.382µs 2024-09-06T14:25:35.900 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.799238722Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-09-06T14:25:35.900 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.799894708Z level=info msg="Migration successfully executed" id="Add unique index alert_rule_tag.alert_id_tag_id" duration=657.833µs 2024-09-06T14:25:35.901 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.801833069Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-09-06T14:25:35.901 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.802351871Z level=info msg="Migration successfully executed" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" duration=518.63µs 2024-09-06T14:25:35.901 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.804509672Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-09-06T14:25:35.901 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.810362965Z level=info msg="Migration successfully executed" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" duration=5.830975ms 2024-09-06T14:25:35.901 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.812842279Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-09-06T14:25:35.901 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.813329582Z level=info msg="Migration successfully executed" id="Create alert_rule_tag table v2" duration=487.315µs 2024-09-06T14:25:35.901 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.813680975Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-09-06T14:25:35.901 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.814227019Z level=info msg="Migration successfully executed" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" duration=545.653µs 2024-09-06T14:25:35.901 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.81457621Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-09-06T14:25:35.901 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.814769617Z level=info msg="Migration successfully executed" id="copy alert_rule_tag v1 to v2" duration=194.602µs 2024-09-06T14:25:35.901 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.817097818Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-09-06T14:25:35.901 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.817461693Z level=info msg="Migration successfully executed" id="drop table alert_rule_tag_v1" duration=364.246µs 2024-09-06T14:25:35.901 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.81781421Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-09-06T14:25:35.902 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.818296738Z level=info msg="Migration successfully executed" id="create alert_notification table v1" duration=482.853µs 2024-09-06T14:25:35.902 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.820617332Z level=info msg="Executing migration" id="Add column is_default" 2024-09-06T14:25:35.902 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.822736455Z level=info msg="Migration successfully executed" id="Add column is_default" duration=2.119694ms 2024-09-06T14:25:35.902 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.823104472Z level=info msg="Executing migration" id="Add column frequency" 2024-09-06T14:25:35.902 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.825284013Z level=info msg="Migration successfully executed" id="Add column frequency" duration=2.179956ms 2024-09-06T14:25:35.902 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.825894744Z level=info msg="Executing migration" id="Add column send_reminder" 2024-09-06T14:25:35.902 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.828231016Z level=info msg="Migration successfully executed" id="Add column send_reminder" duration=2.336001ms 2024-09-06T14:25:35.902 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.828590144Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-09-06T14:25:35.903 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.830672325Z level=info msg="Migration successfully executed" id="Add column disable_resolve_message" duration=2.081727ms 2024-09-06T14:25:35.903 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.831068412Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-09-06T14:25:35.903 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.831602169Z level=info msg="Migration successfully executed" id="add index alert_notification org_id & name" duration=533.548µs 2024-09-06T14:25:35.903 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.831948705Z level=info msg="Executing migration" id="Update alert table charset" 2024-09-06T14:25:35.903 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.831963662Z level=info msg="Migration successfully executed" id="Update alert table charset" duration=15.591µs 2024-09-06T14:25:35.903 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.834527875Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-09-06T14:25:35.903 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.834562462Z level=info msg="Migration successfully executed" id="Update alert_notification table charset" duration=35.708µs 2024-09-06T14:25:35.903 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.834962275Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-09-06T14:25:35.903 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.835449919Z level=info msg="Migration successfully executed" id="create notification_journal table v1" duration=487.945µs 2024-09-06T14:25:35.903 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.835870407Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-09-06T14:25:35.903 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.836448297Z level=info msg="Migration successfully executed" id="add index notification_journal org_id & alert_id & notifier_id" duration=577.699µs 2024-09-06T14:25:35.904 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.836818574Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-09-06T14:25:35.904 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.837354374Z level=info msg="Migration successfully executed" id="drop alert_notification_journal" duration=535.628µs 2024-09-06T14:25:35.904 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.839544906Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-09-06T14:25:35.904 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.84005515Z level=info msg="Migration successfully executed" id="create alert_notification_state table v1" duration=516.703µs 2024-09-06T14:25:35.904 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.842238948Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-09-06T14:25:35.904 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.842801356Z level=info msg="Migration successfully executed" id="add index alert_notification_state org_id & alert_id & notifier_id" duration=562.686µs 2024-09-06T14:25:35.904 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.843158004Z level=info msg="Executing migration" id="Add for to alert table" 2024-09-06T14:25:35.904 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.845287591Z level=info msg="Migration successfully executed" id="Add for to alert table" duration=2.129381ms 2024-09-06T14:25:35.904 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.845653642Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-09-06T14:25:35.904 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.847886595Z level=info msg="Migration successfully executed" id="Add column uid in alert_notification" duration=2.232477ms 2024-09-06T14:25:35.904 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.848248135Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-09-06T14:25:35.904 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.848366153Z level=info msg="Migration successfully executed" id="Update uid column values in alert_notification" duration=118.171µs 2024-09-06T14:25:35.904 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.849068193Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-09-06T14:25:35.904 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.84961156Z level=info msg="Migration successfully executed" id="Add unique index alert_notification_org_id_uid" duration=543.395µs 2024-09-06T14:25:35.905 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.84995865Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-09-06T14:25:35.905 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.850500699Z level=info msg="Migration successfully executed" id="Remove unique index org_id_name" duration=541.868µs 2024-09-06T14:25:35.905 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.850933394Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-09-06T14:25:35.905 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.853117212Z level=info msg="Migration successfully executed" id="Add column secure_settings in alert_notification" duration=2.183989ms 2024-09-06T14:25:35.905 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.853486318Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-09-06T14:25:35.905 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.853528365Z level=info msg="Migration successfully executed" id="alter alert.settings to mediumtext" duration=44.184µs 2024-09-06T14:25:35.905 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.854329499Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-09-06T14:25:35.905 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.854889438Z level=info msg="Migration successfully executed" id="Add non-unique index alert_notification_state_alert_id" duration=560.132µs 2024-09-06T14:25:35.905 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.855245164Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-09-06T14:25:35.905 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.855874424Z level=info msg="Migration successfully executed" id="Add non-unique index alert_rule_tag_alert_id" duration=628.929µs 2024-09-06T14:25:35.905 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.856248565Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-09-06T14:25:35.905 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.856306765Z level=info msg="Migration successfully executed" id="Drop old annotation table v4" duration=58.296µs 2024-09-06T14:25:35.906 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.857188558Z level=info msg="Executing migration" id="create annotation table v5" 2024-09-06T14:25:35.906 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.857806907Z level=info msg="Migration successfully executed" id="create annotation table v5" duration=617.165µs 2024-09-06T14:25:35.906 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.858174738Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-09-06T14:25:35.906 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.858713727Z level=info msg="Migration successfully executed" id="add index annotation 0 v3" duration=539.305µs 2024-09-06T14:25:35.906 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.859092133Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-09-06T14:25:35.906 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.859630398Z level=info msg="Migration successfully executed" id="add index annotation 1 v3" duration=538.024µs 2024-09-06T14:25:35.906 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.859978237Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-09-06T14:25:35.906 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.860515892Z level=info msg="Migration successfully executed" id="add index annotation 2 v3" duration=537.305µs 2024-09-06T14:25:35.906 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.860868184Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-09-06T14:25:35.906 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.861459753Z level=info msg="Migration successfully executed" id="add index annotation 3 v3" duration=591.471µs 2024-09-06T14:25:35.906 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.863605454Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-09-06T14:25:35.907 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.864225534Z level=info msg="Migration successfully executed" id="add index annotation 4 v3" duration=620.274µs 2024-09-06T14:25:35.907 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.866652249Z level=info msg="Executing migration" id="Update annotation table charset" 2024-09-06T14:25:35.907 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.866693255Z level=info msg="Migration successfully executed" id="Update annotation table charset" duration=42.479µs 2024-09-06T14:25:35.907 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.86722267Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-09-06T14:25:35.907 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.869640629Z level=info msg="Migration successfully executed" id="Add column region_id to annotation table" duration=2.417293ms 2024-09-06T14:25:35.907 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.870082235Z level=info msg="Executing migration" id="Drop category_id index" 2024-09-06T14:25:35.907 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.870623577Z level=info msg="Migration successfully executed" id="Drop category_id index" duration=541.241µs 2024-09-06T14:25:35.907 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.872953571Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-09-06T14:25:35.907 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.875259197Z level=info msg="Migration successfully executed" id="Add column tags to annotation table" duration=2.305376ms 2024-09-06T14:25:35.907 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.875622491Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-09-06T14:25:35.907 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.876062789Z level=info msg="Migration successfully executed" id="Create annotation_tag table v2" duration=441.114µs 2024-09-06T14:25:35.907 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.876411594Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-09-06T14:25:35.908 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.876971494Z level=info msg="Migration successfully executed" id="Add unique index annotation_tag.annotation_id_tag_id" duration=559.606µs 2024-09-06T14:25:35.908 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.879228164Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-09-06T14:25:35.908 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.879783864Z level=info msg="Migration successfully executed" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" duration=555.62µs 2024-09-06T14:25:35.908 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.882320427Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-09-06T14:25:35.998 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:25:36.141 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.890614616Z level=info msg="Migration successfully executed" id="Rename table annotation_tag to annotation_tag_v2 - v2" duration=8.295618ms 2024-09-06T14:25:36.141 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.890997279Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-09-06T14:25:36.141 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.891546663Z level=info msg="Migration successfully executed" id="Create annotation_tag table v3" duration=549.379µs 2024-09-06T14:25:36.141 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.891952412Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-09-06T14:25:36.141 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.892613013Z level=info msg="Migration successfully executed" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" duration=661.098µs 2024-09-06T14:25:36.142 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.895001484Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-09-06T14:25:36.142 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.895231058Z level=info msg="Migration successfully executed" id="copy annotation_tag v2 to v3" duration=229.575µs 2024-09-06T14:25:36.142 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.897805753Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-09-06T14:25:36.142 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.898236642Z level=info msg="Migration successfully executed" id="drop table annotation_tag_v2" duration=431.025µs 2024-09-06T14:25:36.142 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.90060062Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-09-06T14:25:36.142 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.900713Z level=info msg="Migration successfully executed" id="Update alert annotations and set TEXT to empty" duration=111.629µs 2024-09-06T14:25:36.142 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.901095457Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-09-06T14:25:36.142 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.903363182Z level=info msg="Migration successfully executed" id="Add created time to annotation table" duration=2.267415ms 2024-09-06T14:25:36.142 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.903818114Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-09-06T14:25:36.142 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.906152752Z level=info msg="Migration successfully executed" id="Add updated time to annotation table" duration=2.334211ms 2024-09-06T14:25:36.142 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.906557285Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-09-06T14:25:36.142 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.907092467Z level=info msg="Migration successfully executed" id="Add index for created in annotation table" duration=534.891µs 2024-09-06T14:25:36.143 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.909442216Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-09-06T14:25:36.143 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.90999931Z level=info msg="Migration successfully executed" id="Add index for updated in annotation table" duration=556.682µs 2024-09-06T14:25:36.143 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.912152792Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-09-06T14:25:36.143 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.912309617Z level=info msg="Migration successfully executed" id="Convert existing annotations from seconds to milliseconds" duration=171.816µs 2024-09-06T14:25:36.143 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.914653485Z level=info msg="Executing migration" id="Add epoch_end column" 2024-09-06T14:25:36.144 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.917019451Z level=info msg="Migration successfully executed" id="Add epoch_end column" duration=2.365635ms 2024-09-06T14:25:36.144 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.917436373Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-09-06T14:25:36.144 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.91801551Z level=info msg="Migration successfully executed" id="Add index for epoch_end" duration=578.574µs 2024-09-06T14:25:36.144 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.920026147Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-09-06T14:25:36.144 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.920152568Z level=info msg="Migration successfully executed" id="Make epoch_end the same as epoch" duration=126.487µs 2024-09-06T14:25:36.144 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.920551942Z level=info msg="Executing migration" id="Move region to single row" 2024-09-06T14:25:36.144 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.920764742Z level=info msg="Migration successfully executed" id="Move region to single row" duration=215.582µs 2024-09-06T14:25:36.144 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.92322452Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-09-06T14:25:36.144 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.923894397Z level=info msg="Migration successfully executed" id="Remove index org_id_epoch from annotation table" duration=671.216µs 2024-09-06T14:25:36.144 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.925730535Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-09-06T14:25:36.144 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.926245195Z level=info msg="Migration successfully executed" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" duration=514.499µs 2024-09-06T14:25:36.144 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.928393655Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-09-06T14:25:36.144 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.928969266Z level=info msg="Migration successfully executed" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" duration=575.298µs 2024-09-06T14:25:36.145 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.929322917Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-09-06T14:25:36.145 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.929867962Z level=info msg="Migration successfully executed" id="Add index for org_id_epoch_end_epoch on annotation table" duration=544.691µs 2024-09-06T14:25:36.145 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.932028665Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-09-06T14:25:36.145 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.932649848Z level=info msg="Migration successfully executed" id="Remove index org_id_epoch_epoch_end from annotation table" duration=623.399µs 2024-09-06T14:25:36.145 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.934415645Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-09-06T14:25:36.145 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.934968352Z level=info msg="Migration successfully executed" id="Add index for alert_id on annotation table" duration=552.393µs 2024-09-06T14:25:36.145 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.937242691Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-09-06T14:25:36.145 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.937299057Z level=info msg="Migration successfully executed" id="Increase tags column to length 4096" duration=56.583µs 2024-09-06T14:25:36.145 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.937722044Z level=info msg="Executing migration" id="create test_data table" 2024-09-06T14:25:36.145 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.938237902Z level=info msg="Migration successfully executed" id="create test_data table" duration=515.454µs 2024-09-06T14:25:36.145 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.940384728Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-09-06T14:25:36.145 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.940932676Z level=info msg="Migration successfully executed" id="create dashboard_version table v1" duration=548.451µs 2024-09-06T14:25:36.146 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.943059596Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-09-06T14:25:36.146 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.943612078Z level=info msg="Migration successfully executed" id="add index dashboard_version.dashboard_id" duration=552.403µs 2024-09-06T14:25:36.146 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.945744538Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-09-06T14:25:36.146 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.946320785Z level=info msg="Migration successfully executed" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" duration=576.557µs 2024-09-06T14:25:36.146 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.946675401Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-09-06T14:25:36.146 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.946793488Z level=info msg="Migration successfully executed" id="Set dashboard version to 1 where 0" duration=117.09µs 2024-09-06T14:25:36.146 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.947716395Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-09-06T14:25:36.146 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.948056746Z level=info msg="Migration successfully executed" id="save existing dashboard data in dashboard_version table v1" duration=343.431µs 2024-09-06T14:25:36.146 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.948456205Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-09-06T14:25:36.146 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.948510991Z level=info msg="Migration successfully executed" id="alter dashboard_version.data to mediumtext v1" duration=54.4µs 2024-09-06T14:25:36.146 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.94929918Z level=info msg="Executing migration" id="create team table" 2024-09-06T14:25:36.146 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.949825445Z level=info msg="Migration successfully executed" id="create team table" duration=526.017µs 2024-09-06T14:25:36.146 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.950216578Z level=info msg="Executing migration" id="add index team.org_id" 2024-09-06T14:25:36.147 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.950872494Z level=info msg="Migration successfully executed" id="add index team.org_id" duration=655.68µs 2024-09-06T14:25:36.147 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.951267323Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-09-06T14:25:36.147 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.951858535Z level=info msg="Migration successfully executed" id="add unique index team_org_id_name" duration=591.054µs 2024-09-06T14:25:36.147 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.952255788Z level=info msg="Executing migration" id="Add column uid in team" 2024-09-06T14:25:36.147 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.954820498Z level=info msg="Migration successfully executed" id="Add column uid in team" duration=2.564325ms 2024-09-06T14:25:36.147 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.955223551Z level=info msg="Executing migration" id="Update uid column values in team" 2024-09-06T14:25:36.147 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.955354606Z level=info msg="Migration successfully executed" id="Update uid column values in team" duration=132.54µs 2024-09-06T14:25:36.147 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.955775789Z level=info msg="Executing migration" id="Add unique index team_org_id_uid" 2024-09-06T14:25:36.147 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.956359784Z level=info msg="Migration successfully executed" id="Add unique index team_org_id_uid" duration=583.927µs 2024-09-06T14:25:36.148 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.956754152Z level=info msg="Executing migration" id="create team member table" 2024-09-06T14:25:36.149 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.95727004Z level=info msg="Migration successfully executed" id="create team member table" duration=515.65µs 2024-09-06T14:25:36.149 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.957641024Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-09-06T14:25:36.149 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.958205337Z level=info msg="Migration successfully executed" id="add index team_member.org_id" duration=564.22µs 2024-09-06T14:25:36.149 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.958590025Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-09-06T14:25:36.149 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.959167469Z level=info msg="Migration successfully executed" id="add unique index team_member_org_id_team_id_user_id" duration=577.216µs 2024-09-06T14:25:36.149 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.959575647Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-09-06T14:25:36.149 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.960140335Z level=info msg="Migration successfully executed" id="add index team_member.team_id" duration=564.436µs 2024-09-06T14:25:36.149 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.960562851Z level=info msg="Executing migration" id="Add column email to team table" 2024-09-06T14:25:36.149 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.9632964Z level=info msg="Migration successfully executed" id="Add column email to team table" duration=2.73171ms 2024-09-06T14:25:36.149 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.963681453Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-09-06T14:25:36.149 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.96647774Z level=info msg="Migration successfully executed" id="Add column external to team_member table" duration=2.795978ms 2024-09-06T14:25:36.149 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.96688835Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-09-06T14:25:36.150 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.969513055Z level=info msg="Migration successfully executed" id="Add column permission to team_member table" duration=2.6245ms 2024-09-06T14:25:36.150 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.969892604Z level=info msg="Executing migration" id="create dashboard acl table" 2024-09-06T14:25:36.150 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.970568484Z level=info msg="Migration successfully executed" id="create dashboard acl table" duration=674.932µs 2024-09-06T14:25:36.150 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.970963361Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-09-06T14:25:36.150 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.971561896Z level=info msg="Migration successfully executed" id="add index dashboard_acl_dashboard_id" duration=599.101µs 2024-09-06T14:25:36.150 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.971955896Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-09-06T14:25:36.150 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.972635234Z level=info msg="Migration successfully executed" id="add unique index dashboard_acl_dashboard_id_user_id" duration=679.035µs 2024-09-06T14:25:36.150 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.973024215Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-09-06T14:25:36.150 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.973636604Z level=info msg="Migration successfully executed" id="add unique index dashboard_acl_dashboard_id_team_id" duration=612.51µs 2024-09-06T14:25:36.150 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.97399586Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-09-06T14:25:36.150 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.974577614Z level=info msg="Migration successfully executed" id="add index dashboard_acl_user_id" duration=583.055µs 2024-09-06T14:25:36.150 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.974934972Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-09-06T14:25:36.150 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.975520838Z level=info msg="Migration successfully executed" id="add index dashboard_acl_team_id" duration=584.29µs 2024-09-06T14:25:36.151 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.975881402Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-09-06T14:25:36.151 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.976484349Z level=info msg="Migration successfully executed" id="add index dashboard_acl_org_id_role" duration=602.47µs 2024-09-06T14:25:36.151 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.976864741Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-09-06T14:25:36.151 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.977481762Z level=info msg="Migration successfully executed" id="add index dashboard_permission" duration=616.776µs 2024-09-06T14:25:36.151 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.977865827Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-09-06T14:25:36.151 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.978222849Z level=info msg="Migration successfully executed" id="save default acl rules in dashboard_acl table" duration=356.846µs 2024-09-06T14:25:36.151 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.978698409Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-09-06T14:25:36.151 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.978857597Z level=info msg="Migration successfully executed" id="delete acl rules for deleted dashboards and folders" duration=159.068µs 2024-09-06T14:25:36.151 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.97956903Z level=info msg="Executing migration" id="create tag table" 2024-09-06T14:25:36.151 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.980103077Z level=info msg="Migration successfully executed" id="create tag table" duration=533.998µs 2024-09-06T14:25:36.151 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.980521062Z level=info msg="Executing migration" id="add index tag.key_value" 2024-09-06T14:25:36.151 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.981106511Z level=info msg="Migration successfully executed" id="add index tag.key_value" duration=583.72µs 2024-09-06T14:25:36.151 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.981509434Z level=info msg="Executing migration" id="create login attempt table" 2024-09-06T14:25:36.151 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.982015494Z level=info msg="Migration successfully executed" id="create login attempt table" duration=505.889µs 2024-09-06T14:25:36.152 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.982411436Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-09-06T14:25:36.152 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.983000605Z level=info msg="Migration successfully executed" id="add index login_attempt.username" duration=589.107µs 2024-09-06T14:25:36.152 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.983395137Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-09-06T14:25:36.152 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.983979592Z level=info msg="Migration successfully executed" id="drop index IDX_login_attempt_username - v1" duration=584.312µs 2024-09-06T14:25:36.152 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.984390969Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-09-06T14:25:36.152 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.992250222Z level=info msg="Migration successfully executed" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" duration=7.858945ms 2024-09-06T14:25:36.152 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.992671184Z level=info msg="Executing migration" id="create login_attempt v2" 2024-09-06T14:25:36.152 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.993166505Z level=info msg="Migration successfully executed" id="create login_attempt v2" duration=495.047µs 2024-09-06T14:25:36.152 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.993550037Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-09-06T14:25:36.152 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.994117697Z level=info msg="Migration successfully executed" id="create index IDX_login_attempt_username - v2" duration=563.655µs 2024-09-06T14:25:36.152 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.994483801Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-09-06T14:25:36.152 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.994703067Z level=info msg="Migration successfully executed" id="copy login_attempt v1 to v2" duration=219.14µs 2024-09-06T14:25:36.152 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.995518855Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-09-06T14:25:36.152 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.996050344Z level=info msg="Migration successfully executed" id="drop login_attempt_tmp_qwerty" duration=533.802µs 2024-09-06T14:25:36.153 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.998238638Z level=info msg="Executing migration" id="create user auth table" 2024-09-06T14:25:36.153 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:35 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:35.998807153Z level=info msg="Migration successfully executed" id="create user auth table" duration=570.981µs 2024-09-06T14:25:36.153 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.0010879Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-09-06T14:25:36.153 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.001716804Z level=info msg="Migration successfully executed" id="create index IDX_user_auth_auth_module_auth_id - v1" duration=628.694µs 2024-09-06T14:25:36.153 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.004056751Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-09-06T14:25:36.153 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.004165798Z level=info msg="Migration successfully executed" id="alter user_auth.auth_id to length 190" duration=111.831µs 2024-09-06T14:25:36.153 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.006992116Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-09-06T14:25:36.153 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.010052476Z level=info msg="Migration successfully executed" id="Add OAuth access token to user_auth" duration=3.061887ms 2024-09-06T14:25:36.153 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.010415163Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-09-06T14:25:36.153 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.01331967Z level=info msg="Migration successfully executed" id="Add OAuth refresh token to user_auth" duration=2.904191ms 2024-09-06T14:25:36.153 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.013677908Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-09-06T14:25:36.153 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.016582313Z level=info msg="Migration successfully executed" id="Add OAuth token type to user_auth" duration=2.902312ms 2024-09-06T14:25:36.153 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.016917826Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-09-06T14:25:36.153 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.019905506Z level=info msg="Migration successfully executed" id="Add OAuth expiry to user_auth" duration=2.987345ms 2024-09-06T14:25:36.153 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.020270907Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-09-06T14:25:36.154 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.020872317Z level=info msg="Migration successfully executed" id="Add index to user_id column in user_auth" duration=602.069µs 2024-09-06T14:25:36.154 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.023333935Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-09-06T14:25:36.154 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.026448027Z level=info msg="Migration successfully executed" id="Add OAuth ID token to user_auth" duration=3.117493ms 2024-09-06T14:25:36.154 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.026811412Z level=info msg="Executing migration" id="create server_lock table" 2024-09-06T14:25:36.154 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.027320042Z level=info msg="Migration successfully executed" id="create server_lock table" duration=508.343µs 2024-09-06T14:25:36.154 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.029522853Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-09-06T14:25:36.154 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.030096397Z level=info msg="Migration successfully executed" id="add index server_lock.operation_uid" duration=573.199µs 2024-09-06T14:25:36.154 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.032439416Z level=info msg="Executing migration" id="create user auth token table" 2024-09-06T14:25:36.154 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.03331076Z level=info msg="Migration successfully executed" id="create user auth token table" duration=871.621µs 2024-09-06T14:25:36.154 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.035238191Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-09-06T14:25:36.154 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.036007662Z level=info msg="Migration successfully executed" id="add unique index user_auth_token.auth_token" duration=770.23µs 2024-09-06T14:25:36.154 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.037916407Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-09-06T14:25:36.154 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.038555642Z level=info msg="Migration successfully executed" id="add unique index user_auth_token.prev_auth_token" duration=639.165µs 2024-09-06T14:25:36.154 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.040806038Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-09-06T14:25:36.155 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.041525593Z level=info msg="Migration successfully executed" id="add index user_auth_token.user_id" duration=719.842µs 2024-09-06T14:25:36.155 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.043692358Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-09-06T14:25:36.155 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.046885152Z level=info msg="Migration successfully executed" id="Add revoked_at to the user auth token" duration=3.192189ms 2024-09-06T14:25:36.155 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.047284407Z level=info msg="Executing migration" id="add index user_auth_token.revoked_at" 2024-09-06T14:25:36.155 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.047914928Z level=info msg="Migration successfully executed" id="add index user_auth_token.revoked_at" duration=630.558µs 2024-09-06T14:25:36.155 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.050098346Z level=info msg="Executing migration" id="create cache_data table" 2024-09-06T14:25:36.155 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.050702514Z level=info msg="Migration successfully executed" id="create cache_data table" duration=603.229µs 2024-09-06T14:25:36.155 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.052828113Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-09-06T14:25:36.155 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.053421383Z level=info msg="Migration successfully executed" id="add unique index cache_data.cache_key" duration=592.835µs 2024-09-06T14:25:36.155 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.055787428Z level=info msg="Executing migration" id="create short_url table v1" 2024-09-06T14:25:36.155 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.056370104Z level=info msg="Migration successfully executed" id="create short_url table v1" duration=580.552µs 2024-09-06T14:25:36.155 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.058524195Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-09-06T14:25:36.155 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.059139104Z level=info msg="Migration successfully executed" id="add index short_url.org_id-uid" duration=614.709µs 2024-09-06T14:25:36.156 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.061158117Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-09-06T14:25:36.156 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.061226197Z level=info msg="Migration successfully executed" id="alter table short_url alter column created_by type to bigint" duration=68.363µs 2024-09-06T14:25:36.156 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.063774588Z level=info msg="Executing migration" id="delete alert_definition table" 2024-09-06T14:25:36.156 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.063851958Z level=info msg="Migration successfully executed" id="delete alert_definition table" duration=77.797µs 2024-09-06T14:25:36.156 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.066390825Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-09-06T14:25:36.156 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.066967724Z level=info msg="Migration successfully executed" id="recreate alert_definition table" duration=575.667µs 2024-09-06T14:25:36.156 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.069129706Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-09-06T14:25:36.156 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.06978852Z level=info msg="Migration successfully executed" id="add index in alert_definition on org_id and title columns" duration=658.509µs 2024-09-06T14:25:36.156 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.071854252Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-09-06T14:25:36.156 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.072491439Z level=info msg="Migration successfully executed" id="add index in alert_definition on org_id and uid columns" duration=638.133µs 2024-09-06T14:25:36.156 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.074446602Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-09-06T14:25:36.156 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.074521511Z level=info msg="Migration successfully executed" id="alter alert_definition table data column to mediumtext in mysql" duration=75.097µs 2024-09-06T14:25:36.156 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.077131606Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-09-06T14:25:36.156 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.077722796Z level=info msg="Migration successfully executed" id="drop index in alert_definition on org_id and title columns" duration=591.045µs 2024-09-06T14:25:36.157 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.079693337Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-09-06T14:25:36.157 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.080263878Z level=info msg="Migration successfully executed" id="drop index in alert_definition on org_id and uid columns" duration=570.307µs 2024-09-06T14:25:36.157 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.08241591Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-09-06T14:25:36.157 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.083067774Z level=info msg="Migration successfully executed" id="add unique index in alert_definition on org_id and title columns" duration=651.391µs 2024-09-06T14:25:36.157 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.085154354Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-09-06T14:25:36.157 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.085826858Z level=info msg="Migration successfully executed" id="add unique index in alert_definition on org_id and uid columns" duration=674.462µs 2024-09-06T14:25:36.157 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.087746562Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-09-06T14:25:36.157 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.091235573Z level=info msg="Migration successfully executed" id="Add column paused in alert_definition" duration=3.490831ms 2024-09-06T14:25:36.157 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.091615475Z level=info msg="Executing migration" id="drop alert_definition table" 2024-09-06T14:25:36.157 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.09226184Z level=info msg="Migration successfully executed" id="drop alert_definition table" duration=646.142µs 2024-09-06T14:25:36.157 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.094317521Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-09-06T14:25:36.157 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.094405102Z level=info msg="Migration successfully executed" id="delete alert_definition_version table" duration=87.748µs 2024-09-06T14:25:36.157 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.096794622Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-09-06T14:25:36.158 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.09737578Z level=info msg="Migration successfully executed" id="recreate alert_definition_version table" duration=580.785µs 2024-09-06T14:25:36.158 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.097740828Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-09-06T14:25:36.158 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.098355173Z level=info msg="Migration successfully executed" id="add index in alert_definition_version table on alert_definition_id and version columns" duration=614.097µs 2024-09-06T14:25:36.158 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.098712396Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-09-06T14:25:36.158 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.099328336Z level=info msg="Migration successfully executed" id="add index in alert_definition_version table on alert_definition_uid and version columns" duration=615.606µs 2024-09-06T14:25:36.158 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.099687963Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-09-06T14:25:36.158 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.099736676Z level=info msg="Migration successfully executed" id="alter alert_definition_version table data column to mediumtext in mysql" duration=48.785µs 2024-09-06T14:25:36.158 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.100720394Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-09-06T14:25:36.158 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.101473294Z level=info msg="Migration successfully executed" id="drop alert_definition_version table" duration=748.379µs 2024-09-06T14:25:36.158 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.101831256Z level=info msg="Executing migration" id="create alert_instance table" 2024-09-06T14:25:36.158 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.10243298Z level=info msg="Migration successfully executed" id="create alert_instance table" duration=601.358µs 2024-09-06T14:25:36.158 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.10279846Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-09-06T14:25:36.158 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.103409722Z level=info msg="Migration successfully executed" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" duration=612.138µs 2024-09-06T14:25:36.158 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.103772973Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-09-06T14:25:36.159 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.104372954Z level=info msg="Migration successfully executed" id="add index in alert_instance table on def_org_id, current_state columns" duration=599.45µs 2024-09-06T14:25:36.159 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.104745796Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-09-06T14:25:36.159 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.108207143Z level=info msg="Migration successfully executed" id="add column current_state_end to alert_instance" duration=3.461036ms 2024-09-06T14:25:36.159 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.108578693Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-09-06T14:25:36.159 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.109155567Z level=info msg="Migration successfully executed" id="remove index def_org_id, def_uid, current_state on alert_instance" duration=576.848µs 2024-09-06T14:25:36.159 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.109512982Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-09-06T14:25:36.159 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.110067695Z level=info msg="Migration successfully executed" id="remove index def_org_id, current_state on alert_instance" duration=554.615µs 2024-09-06T14:25:36.159 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.110400487Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-09-06T14:25:36.159 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.127899435Z level=info msg="Migration successfully executed" id="rename def_org_id to rule_org_id in alert_instance" duration=17.500144ms 2024-09-06T14:25:36.159 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.1283009Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-09-06T14:25:36.160 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.144644104Z level=info msg="Migration successfully executed" id="rename def_uid to rule_uid in alert_instance" duration=16.342964ms 2024-09-06T14:25:36.160 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.145037267Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-09-06T14:25:36.160 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.14572171Z level=info msg="Migration successfully executed" id="add index rule_org_id, rule_uid, current_state on alert_instance" duration=684.284µs 2024-09-06T14:25:36.160 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.146088127Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-09-06T14:25:36.160 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.146749054Z level=info msg="Migration successfully executed" id="add index rule_org_id, current_state on alert_instance" duration=660.181µs 2024-09-06T14:25:36.160 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.14711894Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-09-06T14:25:36.160 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.150355428Z level=info msg="Migration successfully executed" id="add current_reason column related to current_state" duration=3.236098ms 2024-09-06T14:25:36.160 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.15070481Z level=info msg="Executing migration" id="add result_fingerprint column to alert_instance" 2024-09-06T14:25:36.392 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.153915466Z level=info msg="Migration successfully executed" id="add result_fingerprint column to alert_instance" duration=3.210384ms 2024-09-06T14:25:36.392 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.154267051Z level=info msg="Executing migration" id="create alert_rule table" 2024-09-06T14:25:36.393 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.154887288Z level=info msg="Migration successfully executed" id="create alert_rule table" duration=620.029µs 2024-09-06T14:25:36.393 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.155264523Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-09-06T14:25:36.393 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.156053459Z level=info msg="Migration successfully executed" id="add index in alert_rule on org_id and title columns" duration=788.486µs 2024-09-06T14:25:36.393 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.156449445Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-09-06T14:25:36.393 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.157133025Z level=info msg="Migration successfully executed" id="add index in alert_rule on org_id and uid columns" duration=683.447µs 2024-09-06T14:25:36.393 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.157491519Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-09-06T14:25:36.393 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.158299899Z level=info msg="Migration successfully executed" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" duration=807.731µs 2024-09-06T14:25:36.393 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.158747589Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-09-06T14:25:36.393 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.158787194Z level=info msg="Migration successfully executed" id="alter alert_rule table data column to mediumtext in mysql" duration=40.024µs 2024-09-06T14:25:36.393 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.159588863Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-09-06T14:25:36.394 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.163039185Z level=info msg="Migration successfully executed" id="add column for to alert_rule" duration=3.450035ms 2024-09-06T14:25:36.394 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.163395389Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-09-06T14:25:36.394 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.166748125Z level=info msg="Migration successfully executed" id="add column annotations to alert_rule" duration=3.352344ms 2024-09-06T14:25:36.394 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.16713959Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-09-06T14:25:36.394 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.170491196Z level=info msg="Migration successfully executed" id="add column labels to alert_rule" duration=3.3514ms 2024-09-06T14:25:36.394 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.170841737Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-09-06T14:25:36.394 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.171426468Z level=info msg="Migration successfully executed" id="remove unique index from alert_rule on org_id, title columns" duration=584.652µs 2024-09-06T14:25:36.394 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.1736309Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-09-06T14:25:36.394 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.174382616Z level=info msg="Migration successfully executed" id="add index in alert_rule on org_id, namespase_uid and title columns" duration=751.412µs 2024-09-06T14:25:36.394 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.176556693Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-09-06T14:25:36.394 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.179933022Z level=info msg="Migration successfully executed" id="add dashboard_uid column to alert_rule" duration=3.375916ms 2024-09-06T14:25:36.394 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.180281481Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-09-06T14:25:36.394 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.183609376Z level=info msg="Migration successfully executed" id="add panel_id column to alert_rule" duration=3.327543ms 2024-09-06T14:25:36.394 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.183957003Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-09-06T14:25:36.395 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.18456913Z level=info msg="Migration successfully executed" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" duration=611.726µs 2024-09-06T14:25:36.395 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.186724935Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-09-06T14:25:36.395 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.190088666Z level=info msg="Migration successfully executed" id="add rule_group_idx column to alert_rule" duration=3.363439ms 2024-09-06T14:25:36.395 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.190447854Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-09-06T14:25:36.395 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.19383005Z level=info msg="Migration successfully executed" id="add is_paused column to alert_rule table" duration=3.38186ms 2024-09-06T14:25:36.395 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.194186539Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-09-06T14:25:36.395 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.194226716Z level=info msg="Migration successfully executed" id="fix is_paused column for alert_rule table" duration=40.452µs 2024-09-06T14:25:36.395 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.197091193Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-09-06T14:25:36.396 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.197821615Z level=info msg="Migration successfully executed" id="create alert_rule_version table" duration=730.198µs 2024-09-06T14:25:36.396 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.198199687Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-09-06T14:25:36.397 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.19888468Z level=info msg="Migration successfully executed" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" duration=685.363µs 2024-09-06T14:25:36.397 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.199588907Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-09-06T14:25:36.397 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.200299853Z level=info msg="Migration successfully executed" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" duration=710.66µs 2024-09-06T14:25:36.397 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.202565356Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-09-06T14:25:36.397 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.202661132Z level=info msg="Migration successfully executed" id="alter alert_rule_version table data column to mediumtext in mysql" duration=97.818µs 2024-09-06T14:25:36.397 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.205235797Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-09-06T14:25:36.397 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.208823325Z level=info msg="Migration successfully executed" id="add column for to alert_rule_version" duration=3.587178ms 2024-09-06T14:25:36.397 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.20918479Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-09-06T14:25:36.397 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.212763039Z level=info msg="Migration successfully executed" id="add column annotations to alert_rule_version" duration=3.57783ms 2024-09-06T14:25:36.397 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.213120095Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-09-06T14:25:36.397 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.216618945Z level=info msg="Migration successfully executed" id="add column labels to alert_rule_version" duration=3.498412ms 2024-09-06T14:25:36.398 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.216959143Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-09-06T14:25:36.398 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.220507702Z level=info msg="Migration successfully executed" id="add rule_group_idx column to alert_rule_version" duration=3.548181ms 2024-09-06T14:25:36.398 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.220868996Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-09-06T14:25:36.398 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.224352433Z level=info msg="Migration successfully executed" id="add is_paused column to alert_rule_versions table" duration=3.483074ms 2024-09-06T14:25:36.398 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.224724399Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-09-06T14:25:36.398 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.224762527Z level=info msg="Migration successfully executed" id="fix is_paused column for alert_rule_version table" duration=38.571µs 2024-09-06T14:25:36.398 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.227735292Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-09-06T14:25:36.398 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.228379754Z level=info msg="Migration successfully executed" id=create_alert_configuration_table duration=648.407µs 2024-09-06T14:25:36.398 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.230557296Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-09-06T14:25:36.398 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.234210972Z level=info msg="Migration successfully executed" id="Add column default in alert_configuration" duration=3.657072ms 2024-09-06T14:25:36.398 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.234570638Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-09-06T14:25:36.398 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.234613768Z level=info msg="Migration successfully executed" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" duration=44.001µs 2024-09-06T14:25:36.399 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.237217724Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-09-06T14:25:36.399 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.240861815Z level=info msg="Migration successfully executed" id="add column org_id in alert_configuration" duration=3.644172ms 2024-09-06T14:25:36.399 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.241220516Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-09-06T14:25:36.399 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.241848105Z level=info msg="Migration successfully executed" id="add index in alert_configuration table on org_id column" duration=627.968µs 2024-09-06T14:25:36.399 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.243963454Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-09-06T14:25:36.399 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.247664247Z level=info msg="Migration successfully executed" id="add configuration_hash column to alert_configuration" duration=3.7018ms 2024-09-06T14:25:36.399 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.248026948Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-09-06T14:25:36.399 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.248548917Z level=info msg="Migration successfully executed" id=create_ngalert_configuration_table duration=521.854µs 2024-09-06T14:25:36.399 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.250790212Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-09-06T14:25:36.399 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.251508803Z level=info msg="Migration successfully executed" id="add index in ngalert_configuration on org_id column" duration=723.428µs 2024-09-06T14:25:36.399 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.25335553Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-09-06T14:25:36.399 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.257039321Z level=info msg="Migration successfully executed" id="add column send_alerts_to in ngalert_configuration" duration=3.683434ms 2024-09-06T14:25:36.399 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.257404932Z level=info msg="Executing migration" id="create provenance_type table" 2024-09-06T14:25:36.400 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.257909753Z level=info msg="Migration successfully executed" id="create provenance_type table" duration=504.656µs 2024-09-06T14:25:36.400 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.260215146Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-09-06T14:25:36.400 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.260881137Z level=info msg="Migration successfully executed" id="add index to uniquify (record_key, record_type, org_id) columns" duration=665.722µs 2024-09-06T14:25:36.400 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.263038111Z level=info msg="Executing migration" id="create alert_image table" 2024-09-06T14:25:36.400 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.263600992Z level=info msg="Migration successfully executed" id="create alert_image table" duration=564.865µs 2024-09-06T14:25:36.400 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.265623775Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-09-06T14:25:36.400 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.266246101Z level=info msg="Migration successfully executed" id="add unique index on token to alert_image table" duration=605.002µs 2024-09-06T14:25:36.400 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.268171173Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-09-06T14:25:36.400 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.268225315Z level=info msg="Migration successfully executed" id="support longer URLs in alert_image table" duration=54.408µs 2024-09-06T14:25:36.400 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.270787719Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-09-06T14:25:36.400 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.271430596Z level=info msg="Migration successfully executed" id=create_alert_configuration_history_table duration=643.037µs 2024-09-06T14:25:36.400 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.273409949Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-09-06T14:25:36.400 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.274029316Z level=info msg="Migration successfully executed" id="drop non-unique orgID index on alert_configuration" duration=619.75µs 2024-09-06T14:25:36.401 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.276199673Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-09-06T14:25:36.401 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.27663753Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-09-06T14:25:36.401 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.278693431Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-09-06T14:25:36.401 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.278959482Z level=info msg="Migration successfully executed" id="extract alertmanager configuration history to separate table" duration=265.826µs 2024-09-06T14:25:36.401 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.279522223Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-09-06T14:25:36.401 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.280111104Z level=info msg="Migration successfully executed" id="add unique index on orgID to alert_configuration" duration=588.563µs 2024-09-06T14:25:36.401 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.280440344Z level=info msg="Executing migration" id="add last_applied column to alert_configuration_history" 2024-09-06T14:25:36.401 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.284146505Z level=info msg="Migration successfully executed" id="add last_applied column to alert_configuration_history" duration=3.705815ms 2024-09-06T14:25:36.401 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.28454796Z level=info msg="Executing migration" id="create library_element table v1" 2024-09-06T14:25:36.401 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.285236205Z level=info msg="Migration successfully executed" id="create library_element table v1" duration=688.137µs 2024-09-06T14:25:36.401 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.285639582Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-09-06T14:25:36.401 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.286346471Z level=info msg="Migration successfully executed" id="add index library_element org_id-folder_id-name-kind" duration=706.746µs 2024-09-06T14:25:36.402 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.286732388Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-09-06T14:25:36.402 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.287282845Z level=info msg="Migration successfully executed" id="create library_element_connection table v1" duration=550.217µs 2024-09-06T14:25:36.402 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.287648882Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-09-06T14:25:36.402 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.288321106Z level=info msg="Migration successfully executed" id="add index library_element_connection element_id-kind-connection_id" duration=671.929µs 2024-09-06T14:25:36.402 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.288704054Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-09-06T14:25:36.402 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.289319168Z level=info msg="Migration successfully executed" id="add unique index library_element org_id_uid" duration=614.769µs 2024-09-06T14:25:36.402 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.289676534Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-09-06T14:25:36.402 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.289695965Z level=info msg="Migration successfully executed" id="increase max description length to 2048" duration=19.435µs 2024-09-06T14:25:36.402 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.290528217Z level=info msg="Executing migration" id="alter library_element model to mediumtext" 2024-09-06T14:25:36.402 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.290575953Z level=info msg="Migration successfully executed" id="alter library_element model to mediumtext" duration=47.946µs 2024-09-06T14:25:36.402 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.291431336Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-09-06T14:25:36.402 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.291774216Z level=info msg="Migration successfully executed" id="clone move dashboard alerts to unified alerting" duration=345.672µs 2024-09-06T14:25:36.402 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.292170733Z level=info msg="Executing migration" id="create data_keys table" 2024-09-06T14:25:36.402 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.292820682Z level=info msg="Migration successfully executed" id="create data_keys table" duration=649.779µs 2024-09-06T14:25:36.403 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.293217576Z level=info msg="Executing migration" id="create secrets table" 2024-09-06T14:25:36.403 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.293774996Z level=info msg="Migration successfully executed" id="create secrets table" duration=557.279µs 2024-09-06T14:25:36.403 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.294130878Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-09-06T14:25:36.403 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.314786329Z level=info msg="Migration successfully executed" id="rename data_keys name column to id" duration=20.652685ms 2024-09-06T14:25:36.403 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.315185661Z level=info msg="Executing migration" id="add name column into data_keys" 2024-09-06T14:25:36.403 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.319195892Z level=info msg="Migration successfully executed" id="add name column into data_keys" duration=4.010068ms 2024-09-06T14:25:36.403 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.319569479Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-09-06T14:25:36.403 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.319675617Z level=info msg="Migration successfully executed" id="copy data_keys id column values into name" duration=107.768µs 2024-09-06T14:25:36.403 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.320055128Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-09-06T14:25:36.403 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.340701695Z level=info msg="Migration successfully executed" id="rename data_keys name column to label" duration=20.646412ms 2024-09-06T14:25:36.403 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.341068963Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-09-06T14:25:36.403 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.361847492Z level=info msg="Migration successfully executed" id="rename data_keys id column back to name" duration=20.778303ms 2024-09-06T14:25:36.403 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.362244136Z level=info msg="Executing migration" id="create kv_store table v1" 2024-09-06T14:25:36.404 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.362827552Z level=info msg="Migration successfully executed" id="create kv_store table v1" duration=584.497µs 2024-09-06T14:25:36.404 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.363189796Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-09-06T14:25:36.404 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.363852177Z level=info msg="Migration successfully executed" id="add index kv_store.org_id-namespace-key" duration=662.001µs 2024-09-06T14:25:36.404 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.364191106Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-09-06T14:25:36.404 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.364324047Z level=info msg="Migration successfully executed" id="update dashboard_uid and panel_id from existing annotations" duration=132.795µs 2024-09-06T14:25:36.404 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.364745822Z level=info msg="Executing migration" id="create permission table" 2024-09-06T14:25:36.404 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.365334336Z level=info msg="Migration successfully executed" id="create permission table" duration=588.139µs 2024-09-06T14:25:36.404 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.365705463Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-09-06T14:25:36.404 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.366313696Z level=info msg="Migration successfully executed" id="add unique index permission.role_id" duration=607.96µs 2024-09-06T14:25:36.404 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.366670991Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-09-06T14:25:36.404 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.367312853Z level=info msg="Migration successfully executed" id="add unique index role_id_action_scope" duration=641.539µs 2024-09-06T14:25:36.404 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.367670794Z level=info msg="Executing migration" id="create role table" 2024-09-06T14:25:36.405 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.36824016Z level=info msg="Migration successfully executed" id="create role table" duration=569.016µs 2024-09-06T14:25:36.405 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.368585076Z level=info msg="Executing migration" id="add column display_name" 2024-09-06T14:25:36.405 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.372937437Z level=info msg="Migration successfully executed" id="add column display_name" duration=4.35213ms 2024-09-06T14:25:36.405 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.373290529Z level=info msg="Executing migration" id="add column group_name" 2024-09-06T14:25:36.405 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.377376513Z level=info msg="Migration successfully executed" id="add column group_name" duration=4.085644ms 2024-09-06T14:25:36.405 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.377745101Z level=info msg="Executing migration" id="add index role.org_id" 2024-09-06T14:25:36.405 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.378386731Z level=info msg="Migration successfully executed" id="add index role.org_id" duration=641.518µs 2024-09-06T14:25:36.405 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.378760897Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-09-06T14:25:36.405 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.379426563Z level=info msg="Migration successfully executed" id="add unique index role_org_id_name" duration=665.292µs 2024-09-06T14:25:36.405 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.379788165Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-09-06T14:25:36.405 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.380436336Z level=info msg="Migration successfully executed" id="add index role_org_id_uid" duration=647.898µs 2024-09-06T14:25:36.405 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.380789579Z level=info msg="Executing migration" id="create team role table" 2024-09-06T14:25:36.405 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.381338026Z level=info msg="Migration successfully executed" id="create team role table" duration=547.994µs 2024-09-06T14:25:36.406 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.381706112Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-09-06T14:25:36.406 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.382371914Z level=info msg="Migration successfully executed" id="add index team_role.org_id" duration=665.598µs 2024-09-06T14:25:36.406 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.38273713Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-09-06T14:25:36.406 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.383429097Z level=info msg="Migration successfully executed" id="add unique index team_role_org_id_team_id_role_id" duration=691.525µs 2024-09-06T14:25:36.406 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.383795057Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-09-06T14:25:36.406 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.384413111Z level=info msg="Migration successfully executed" id="add index team_role.team_id" duration=617.743µs 2024-09-06T14:25:36.406 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.384762619Z level=info msg="Executing migration" id="create user role table" 2024-09-06T14:25:36.406 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.385300709Z level=info msg="Migration successfully executed" id="create user role table" duration=537.7µs 2024-09-06T14:25:36.406 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.385675128Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-09-06T14:25:36.406 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.386312011Z level=info msg="Migration successfully executed" id="add index user_role.org_id" duration=636.621µs 2024-09-06T14:25:36.406 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.386674704Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-09-06T14:25:36.406 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.387314405Z level=info msg="Migration successfully executed" id="add unique index user_role_org_id_user_id_role_id" duration=639.423µs 2024-09-06T14:25:36.406 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.387670119Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-09-06T14:25:36.407 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.388343494Z level=info msg="Migration successfully executed" id="add index user_role.user_id" duration=673.026µs 2024-09-06T14:25:36.407 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.388699924Z level=info msg="Executing migration" id="create builtin role table" 2024-09-06T14:25:36.407 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.389230288Z level=info msg="Migration successfully executed" id="create builtin role table" duration=530.003µs 2024-09-06T14:25:36.407 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.389600692Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-09-06T14:25:36.407 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.390244335Z level=info msg="Migration successfully executed" id="add index builtin_role.role_id" duration=643.476µs 2024-09-06T14:25:36.407 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.390569314Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-09-06T14:25:36.407 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.391221118Z level=info msg="Migration successfully executed" id="add index builtin_role.name" duration=651.62µs 2024-09-06T14:25:36.407 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.391573533Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-09-06T14:25:36.407 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.396204004Z level=info msg="Migration successfully executed" id="Add column org_id to builtin_role table" duration=4.630083ms 2024-09-06T14:25:36.407 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.396584849Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-09-06T14:25:36.407 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.397229648Z level=info msg="Migration successfully executed" id="add index builtin_role.org_id" duration=644.527µs 2024-09-06T14:25:36.407 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.397570379Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-09-06T14:25:36.408 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.398230218Z level=info msg="Migration successfully executed" id="add unique index builtin_role_org_id_role_id_role" duration=659.891µs 2024-09-06T14:25:36.408 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.398566893Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-09-06T14:25:36.408 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.399227546Z level=info msg="Migration successfully executed" id="Remove unique index role_org_id_uid" duration=659.852µs 2024-09-06T14:25:36.408 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.399568214Z level=info msg="Executing migration" id="add unique index role.uid" 2024-09-06T14:25:36.408 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.400219681Z level=info msg="Migration successfully executed" id="add unique index role.uid" duration=651.282µs 2024-09-06T14:25:36.605 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:36.591+0000 7fbf6d90f640 1 -- 172.21.15.142:0/913370885 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbf681073d0 msgr2=0x7fbf68107810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:25:36.605 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:36.591+0000 7fbf6d90f640 1 --2- 172.21.15.142:0/913370885 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbf681073d0 0x7fbf68107810 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fbf5c009900 tx=0x7fbf5c02a410 comp rx=0 tx=0).stop 2024-09-06T14:25:36.605 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:36.591+0000 7fbf6d90f640 1 -- 172.21.15.142:0/913370885 shutdown_connections 2024-09-06T14:25:36.605 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:36.591+0000 7fbf6d90f640 1 --2- 172.21.15.142:0/913370885 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbf68107d50 0x7fbf6810e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:36.605 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:36.591+0000 7fbf6d90f640 1 --2- 172.21.15.142:0/913370885 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbf681073d0 0x7fbf68107810 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:36.605 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:36.591+0000 7fbf6d90f640 1 --2- 172.21.15.142:0/913370885 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbf681019c0 0x7fbf68101da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:36.606 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:36.591+0000 7fbf6d90f640 1 -- 172.21.15.142:0/913370885 >> 172.21.15.142:0/913370885 conn(0x7fbf680fd430 msgr2=0x7fbf680ff850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:25:36.606 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:36.591+0000 7fbf6d90f640 1 -- 172.21.15.142:0/913370885 shutdown_connections 2024-09-06T14:25:36.606 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:36.591+0000 7fbf6d90f640 1 -- 172.21.15.142:0/913370885 wait complete. 2024-09-06T14:25:36.606 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:36.592+0000 7fbf6d90f640 1 Processor -- start 2024-09-06T14:25:36.606 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:36.592+0000 7fbf6d90f640 1 -- start start 2024-09-06T14:25:36.606 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:36.592+0000 7fbf6d90f640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbf681019c0 0x7fbf6819a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:25:36.606 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:36.592+0000 7fbf6d90f640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbf681073d0 0x7fbf6819ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:25:36.606 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:36.592+0000 7fbf6d90f640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbf68107d50 0x7fbf681a0200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:25:36.606 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:36.592+0000 7fbf6d90f640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fbf68112220 con 0x7fbf681019c0 2024-09-06T14:25:36.606 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:36.592+0000 7fbf6d90f640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fbf681120a0 con 0x7fbf68107d50 2024-09-06T14:25:36.606 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:36.592+0000 7fbf6d90f640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7fbf681123a0 con 0x7fbf681073d0 2024-09-06T14:25:36.606 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:36.592+0000 7fbf66ffd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbf681019c0 0x7fbf6819a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:25:36.606 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:36.592+0000 7fbf667fc640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbf681073d0 0x7fbf6819ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:25:36.606 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:36.592+0000 7fbf66ffd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbf681019c0 0x7fbf6819a920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:54624/0 (socket says 172.21.15.142:54624) 2024-09-06T14:25:36.607 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:36.592+0000 7fbf66ffd640 1 -- 172.21.15.142:0/2289133884 learned_addr learned my addr 172.21.15.142:0/2289133884 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:25:36.607 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:36.592+0000 7fbf677fe640 1 --2- 172.21.15.142:0/2289133884 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbf68107d50 0x7fbf681a0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:25:36.607 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:36.593+0000 7fbf66ffd640 1 -- 172.21.15.142:0/2289133884 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbf681073d0 msgr2=0x7fbf6819ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:25:36.607 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:36.593+0000 7fbf66ffd640 1 --2- 172.21.15.142:0/2289133884 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbf681073d0 0x7fbf6819ae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:36.607 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:36.593+0000 7fbf66ffd640 1 -- 172.21.15.142:0/2289133884 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbf68107d50 msgr2=0x7fbf681a0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:25:36.607 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:36.593+0000 7fbf66ffd640 1 --2- 172.21.15.142:0/2289133884 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbf68107d50 0x7fbf681a0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:36.607 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:36.593+0000 7fbf66ffd640 1 -- 172.21.15.142:0/2289133884 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf681a09d0 con 0x7fbf681019c0 2024-09-06T14:25:36.607 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:36.593+0000 7fbf667fc640 1 --2- 172.21.15.142:0/2289133884 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbf681073d0 0x7fbf6819ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-06T14:25:36.607 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:36.593+0000 7fbf66ffd640 1 --2- 172.21.15.142:0/2289133884 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbf681019c0 0x7fbf6819a920 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fbf5800e9f0 tx=0x7fbf5800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:25:36.607 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:36.593+0000 7fbf6c90d640 1 -- 172.21.15.142:0/2289133884 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf5800ccc0 con 0x7fbf681019c0 2024-09-06T14:25:36.607 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:36.593+0000 7fbf6d90f640 1 -- 172.21.15.142:0/2289133884 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf681a0cc0 con 0x7fbf681019c0 2024-09-06T14:25:36.607 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:36.593+0000 7fbf6c90d640 1 -- 172.21.15.142:0/2289133884 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7fbf580108b0 con 0x7fbf681019c0 2024-09-06T14:25:36.607 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:36.593+0000 7fbf6c90d640 1 -- 172.21.15.142:0/2289133884 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf58018640 con 0x7fbf681019c0 2024-09-06T14:25:36.608 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:36.593+0000 7fbf6d90f640 1 -- 172.21.15.142:0/2289133884 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf681a7380 con 0x7fbf681019c0 2024-09-06T14:25:36.608 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:36.594+0000 7fbf6d90f640 1 -- 172.21.15.142:0/2289133884 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf68101da0 con 0x7fbf681019c0 2024-09-06T14:25:36.608 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:36.595+0000 7fbf6c90d640 1 -- 172.21.15.142:0/2289133884 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 22) ==== 99844+0+0 (secure 0 0 0) 0x7fbf58010430 con 0x7fbf681019c0 2024-09-06T14:25:36.608 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:36.595+0000 7fbf6c90d640 1 --2- 172.21.15.142:0/2289133884 >> v2:172.21.15.142:6800/2944545612 conn(0x7fbf50077620 0x7fbf50079ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:25:36.608 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:36.595+0000 7fbf6c90d640 1 -- 172.21.15.142:0/2289133884 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 6043+0+0 (secure 0 0 0) 0x7fbf58099cc0 con 0x7fbf681019c0 2024-09-06T14:25:36.608 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:36.595+0000 7fbf667fc640 1 --2- 172.21.15.142:0/2289133884 >> v2:172.21.15.142:6800/2944545612 conn(0x7fbf50077620 0x7fbf50079ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:25:36.608 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:36.596+0000 7fbf667fc640 1 --2- 172.21.15.142:0/2289133884 >> v2:172.21.15.142:6800/2944545612 conn(0x7fbf50077620 0x7fbf50079ae0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fbf5c031440 tx=0x7fbf5c031850 comp rx=0 tx=0).ready entity=mgr.14601 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:25:36.608 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:36.600+0000 7fbf6c90d640 1 -- 172.21.15.142:0/2289133884 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7fbf58066f50 con 0x7fbf681019c0 2024-09-06T14:25:36.652 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-mon[33787]: pgmap v23: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:25:36.652 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-mon[33787]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:36.653 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-mon[33787]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:36.653 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-mon[33787]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:36.653 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-mon[33787]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:36.653 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-mon[33787]: Deploying daemon node-exporter.a on smithi142 2024-09-06T14:25:36.653 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-mon[33787]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:36.654 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.400569395Z level=info msg="Executing migration" id="create seed assignment table" 2024-09-06T14:25:36.654 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.401054306Z level=info msg="Migration successfully executed" id="create seed assignment table" duration=484.488µs 2024-09-06T14:25:36.654 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.401380742Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-09-06T14:25:36.654 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.402051998Z level=info msg="Migration successfully executed" id="add unique index builtin_role_role_name" duration=671.044µs 2024-09-06T14:25:36.654 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.402383616Z level=info msg="Executing migration" id="add column hidden to role table" 2024-09-06T14:25:36.654 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.407110359Z level=info msg="Migration successfully executed" id="add column hidden to role table" duration=4.726354ms 2024-09-06T14:25:36.654 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.407451065Z level=info msg="Executing migration" id="permission kind migration" 2024-09-06T14:25:36.654 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.412000398Z level=info msg="Migration successfully executed" id="permission kind migration" duration=4.550691ms 2024-09-06T14:25:36.654 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.412361713Z level=info msg="Executing migration" id="permission attribute migration" 2024-09-06T14:25:36.655 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.416873455Z level=info msg="Migration successfully executed" id="permission attribute migration" duration=4.511536ms 2024-09-06T14:25:36.655 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.417222892Z level=info msg="Executing migration" id="permission identifier migration" 2024-09-06T14:25:36.655 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.421723526Z level=info msg="Migration successfully executed" id="permission identifier migration" duration=4.498756ms 2024-09-06T14:25:36.655 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.422059407Z level=info msg="Executing migration" id="add permission identifier index" 2024-09-06T14:25:36.655 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.422758776Z level=info msg="Migration successfully executed" id="add permission identifier index" duration=699.208µs 2024-09-06T14:25:36.655 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.423115223Z level=info msg="Executing migration" id="add permission action scope role_id index" 2024-09-06T14:25:36.655 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.423853829Z level=info msg="Migration successfully executed" id="add permission action scope role_id index" duration=738.178µs 2024-09-06T14:25:36.655 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.424204442Z level=info msg="Executing migration" id="remove permission role_id action scope index" 2024-09-06T14:25:36.655 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.424864259Z level=info msg="Migration successfully executed" id="remove permission role_id action scope index" duration=659.74µs 2024-09-06T14:25:36.655 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.425209993Z level=info msg="Executing migration" id="create query_history table v1" 2024-09-06T14:25:36.655 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.425799343Z level=info msg="Migration successfully executed" id="create query_history table v1" duration=588.993µs 2024-09-06T14:25:36.655 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.426146723Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-09-06T14:25:36.656 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.426822712Z level=info msg="Migration successfully executed" id="add index query_history.org_id-created_by-datasource_uid" duration=675.757µs 2024-09-06T14:25:36.656 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.427165915Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-09-06T14:25:36.656 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.427204364Z level=info msg="Migration successfully executed" id="alter table query_history alter column created_by type to bigint" duration=38.699µs 2024-09-06T14:25:36.656 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.427608072Z level=info msg="Executing migration" id="rbac disabled migrator" 2024-09-06T14:25:36.656 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.427630932Z level=info msg="Migration successfully executed" id="rbac disabled migrator" duration=23.892µs 2024-09-06T14:25:36.656 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.428077018Z level=info msg="Executing migration" id="teams permissions migration" 2024-09-06T14:25:36.656 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.428337943Z level=info msg="Migration successfully executed" id="teams permissions migration" duration=261.241µs 2024-09-06T14:25:36.656 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.428729416Z level=info msg="Executing migration" id="dashboard permissions" 2024-09-06T14:25:36.657 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.42907147Z level=info msg="Migration successfully executed" id="dashboard permissions" duration=342.735µs 2024-09-06T14:25:36.657 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.429453392Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-09-06T14:25:36.657 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.429880809Z level=info msg="Migration successfully executed" id="dashboard permissions uid scopes" duration=427.529µs 2024-09-06T14:25:36.657 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.430317515Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-09-06T14:25:36.657 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.430444808Z level=info msg="Migration successfully executed" id="drop managed folder create actions" duration=127.378µs 2024-09-06T14:25:36.657 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.430851521Z level=info msg="Executing migration" id="alerting notification permissions" 2024-09-06T14:25:36.657 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.431150001Z level=info msg="Migration successfully executed" id="alerting notification permissions" duration=299.699µs 2024-09-06T14:25:36.657 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.431713219Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-09-06T14:25:36.657 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.432244228Z level=info msg="Migration successfully executed" id="create query_history_star table v1" duration=531.129µs 2024-09-06T14:25:36.657 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.432597007Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-09-06T14:25:36.657 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.433262143Z level=info msg="Migration successfully executed" id="add index query_history.user_id-query_uid" duration=665.184µs 2024-09-06T14:25:36.657 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.433637493Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-09-06T14:25:36.658 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.438290794Z level=info msg="Migration successfully executed" id="add column org_id in query_history_star" duration=4.652866ms 2024-09-06T14:25:36.658 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.438661756Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-09-06T14:25:36.658 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.438700276Z level=info msg="Migration successfully executed" id="alter table query_history_star_mig column user_id type to bigint" duration=38.944µs 2024-09-06T14:25:36.658 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.43953957Z level=info msg="Executing migration" id="create correlation table v1" 2024-09-06T14:25:36.658 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.440215001Z level=info msg="Migration successfully executed" id="create correlation table v1" duration=675.604µs 2024-09-06T14:25:36.658 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.44057853Z level=info msg="Executing migration" id="add index correlations.uid" 2024-09-06T14:25:36.658 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.441234607Z level=info msg="Migration successfully executed" id="add index correlations.uid" duration=655.386µs 2024-09-06T14:25:36.658 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.441576948Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-09-06T14:25:36.658 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.442248994Z level=info msg="Migration successfully executed" id="add index correlations.source_uid" duration=670.797µs 2024-09-06T14:25:36.658 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.442609163Z level=info msg="Executing migration" id="add correlation config column" 2024-09-06T14:25:36.659 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.447412637Z level=info msg="Migration successfully executed" id="add correlation config column" duration=4.80384ms 2024-09-06T14:25:36.659 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.447812691Z level=info msg="Executing migration" id="drop index IDX_correlation_uid - v1" 2024-09-06T14:25:36.659 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.448458293Z level=info msg="Migration successfully executed" id="drop index IDX_correlation_uid - v1" duration=645.619µs 2024-09-06T14:25:36.659 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.448807365Z level=info msg="Executing migration" id="drop index IDX_correlation_source_uid - v1" 2024-09-06T14:25:36.659 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.449454071Z level=info msg="Migration successfully executed" id="drop index IDX_correlation_source_uid - v1" duration=645.535µs 2024-09-06T14:25:36.659 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.449804959Z level=info msg="Executing migration" id="Rename table correlation to correlation_tmp_qwerty - v1" 2024-09-06T14:25:36.659 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.463216482Z level=info msg="Migration successfully executed" id="Rename table correlation to correlation_tmp_qwerty - v1" duration=13.411652ms 2024-09-06T14:25:36.659 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.4635926Z level=info msg="Executing migration" id="create correlation v2" 2024-09-06T14:25:36.659 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.464304913Z level=info msg="Migration successfully executed" id="create correlation v2" duration=712.005µs 2024-09-06T14:25:36.659 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.464682783Z level=info msg="Executing migration" id="create index IDX_correlation_uid - v2" 2024-09-06T14:25:36.659 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.465342805Z level=info msg="Migration successfully executed" id="create index IDX_correlation_uid - v2" duration=660.009µs 2024-09-06T14:25:36.659 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.465711528Z level=info msg="Executing migration" id="create index IDX_correlation_source_uid - v2" 2024-09-06T14:25:36.659 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.46641753Z level=info msg="Migration successfully executed" id="create index IDX_correlation_source_uid - v2" duration=705.874µs 2024-09-06T14:25:36.660 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.466782829Z level=info msg="Executing migration" id="create index IDX_correlation_org_id - v2" 2024-09-06T14:25:36.660 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.467438333Z level=info msg="Migration successfully executed" id="create index IDX_correlation_org_id - v2" duration=655.725µs 2024-09-06T14:25:36.660 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.469409064Z level=info msg="Executing migration" id="copy correlation v1 to v2" 2024-09-06T14:25:36.660 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.469596253Z level=info msg="Migration successfully executed" id="copy correlation v1 to v2" duration=187.019µs 2024-09-06T14:25:36.660 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.472133508Z level=info msg="Executing migration" id="drop correlation_tmp_qwerty" 2024-09-06T14:25:36.660 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.472686602Z level=info msg="Migration successfully executed" id="drop correlation_tmp_qwerty" duration=552.789µs 2024-09-06T14:25:36.660 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.473035531Z level=info msg="Executing migration" id="add provisioning column" 2024-09-06T14:25:36.660 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.47813123Z level=info msg="Migration successfully executed" id="add provisioning column" duration=5.095704ms 2024-09-06T14:25:36.660 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.478513549Z level=info msg="Executing migration" id="create entity_events table" 2024-09-06T14:25:36.660 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.479042464Z level=info msg="Migration successfully executed" id="create entity_events table" duration=530.265µs 2024-09-06T14:25:36.660 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.47940715Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-09-06T14:25:36.660 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.48007349Z level=info msg="Migration successfully executed" id="create dashboard public config v1" duration=666.207µs 2024-09-06T14:25:36.660 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.480441011Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-09-06T14:25:36.661 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.480717555Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-09-06T14:25:36.661 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.48132191Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-06T14:25:36.661 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.481603327Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-06T14:25:36.661 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.482099332Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-09-06T14:25:36.661 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.482698748Z level=info msg="Migration successfully executed" id="Drop old dashboard public config table" duration=599.57µs 2024-09-06T14:25:36.661 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.48308325Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-09-06T14:25:36.661 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.483758435Z level=info msg="Migration successfully executed" id="recreate dashboard public config v1" duration=675.27µs 2024-09-06T14:25:36.661 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.484124581Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-09-06T14:25:36.661 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.484811156Z level=info msg="Migration successfully executed" id="create index UQE_dashboard_public_config_uid - v1" duration=686.889µs 2024-09-06T14:25:36.661 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.485179888Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-06T14:25:36.661 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.485878564Z level=info msg="Migration successfully executed" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" duration=698.288µs 2024-09-06T14:25:36.662 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.486220504Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-09-06T14:25:36.662 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.486894588Z level=info msg="Migration successfully executed" id="drop index UQE_dashboard_public_config_uid - v2" duration=673.965µs 2024-09-06T14:25:36.662 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.487245993Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-09-06T14:25:36.662 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.487928182Z level=info msg="Migration successfully executed" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" duration=681.887µs 2024-09-06T14:25:36.662 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.488281239Z level=info msg="Executing migration" id="Drop public config table" 2024-09-06T14:25:36.662 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.488843853Z level=info msg="Migration successfully executed" id="Drop public config table" duration=562.312µs 2024-09-06T14:25:36.663 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.489213107Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-09-06T14:25:36.663 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.489916342Z level=info msg="Migration successfully executed" id="Recreate dashboard public config v2" duration=702.908µs 2024-09-06T14:25:36.663 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.490272943Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-09-06T14:25:36.663 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.490951882Z level=info msg="Migration successfully executed" id="create index UQE_dashboard_public_config_uid - v2" duration=678.725µs 2024-09-06T14:25:36.663 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.491288806Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-09-06T14:25:36.663 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.492006853Z level=info msg="Migration successfully executed" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" duration=718.25µs 2024-09-06T14:25:36.663 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.492358064Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-09-06T14:25:36.663 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.493049423Z level=info msg="Migration successfully executed" id="create index UQE_dashboard_public_config_access_token - v2" duration=690.446µs 2024-09-06T14:25:36.663 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.493536621Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-09-06T14:25:36.663 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.509751759Z level=info msg="Migration successfully executed" id="Rename table dashboard_public_config to dashboard_public - v2" duration=16.214997ms 2024-09-06T14:25:36.663 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.510141487Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-09-06T14:25:36.663 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.515240426Z level=info msg="Migration successfully executed" id="add annotations_enabled column" duration=5.098751ms 2024-09-06T14:25:36.663 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.515619216Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-09-06T14:25:36.664 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.520437428Z level=info msg="Migration successfully executed" id="add time_selection_enabled column" duration=4.817917ms 2024-09-06T14:25:36.664 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.520804041Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-09-06T14:25:36.664 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.52095177Z level=info msg="Migration successfully executed" id="delete orphaned public dashboards" duration=147.813µs 2024-09-06T14:25:36.664 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.521655824Z level=info msg="Executing migration" id="add share column" 2024-09-06T14:25:36.664 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.526425724Z level=info msg="Migration successfully executed" id="add share column" duration=4.769706ms 2024-09-06T14:25:36.664 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.526792896Z level=info msg="Executing migration" id="backfill empty share column fields with default of public" 2024-09-06T14:25:36.664 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.526927163Z level=info msg="Migration successfully executed" id="backfill empty share column fields with default of public" duration=134.425µs 2024-09-06T14:25:36.664 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.527658099Z level=info msg="Executing migration" id="create file table" 2024-09-06T14:25:36.664 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.52825116Z level=info msg="Migration successfully executed" id="create file table" duration=592.916µs 2024-09-06T14:25:36.664 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.528620531Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-09-06T14:25:36.664 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.529288612Z level=info msg="Migration successfully executed" id="file table idx: path natural pk" duration=667.825µs 2024-09-06T14:25:36.664 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.529654113Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-09-06T14:25:36.664 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.530337812Z level=info msg="Migration successfully executed" id="file table idx: parent_folder_path_hash fast folder retrieval" duration=683.446µs 2024-09-06T14:25:36.664 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.530711554Z level=info msg="Executing migration" id="create file_meta table" 2024-09-06T14:25:36.664 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.531210396Z level=info msg="Migration successfully executed" id="create file_meta table" duration=498.677µs 2024-09-06T14:25:36.665 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.531589983Z level=info msg="Executing migration" id="file table idx: path key" 2024-09-06T14:25:36.665 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.532268758Z level=info msg="Migration successfully executed" id="file table idx: path key" duration=678.349µs 2024-09-06T14:25:36.665 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.532655796Z level=info msg="Executing migration" id="set path collation in file table" 2024-09-06T14:25:36.665 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.532694896Z level=info msg="Migration successfully executed" id="set path collation in file table" duration=39.307µs 2024-09-06T14:25:36.665 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.533616935Z level=info msg="Executing migration" id="migrate contents column to mediumblob for MySQL" 2024-09-06T14:25:36.665 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.533736023Z level=info msg="Migration successfully executed" id="migrate contents column to mediumblob for MySQL" duration=125.781µs 2024-09-06T14:25:36.665 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.534317614Z level=info msg="Executing migration" id="managed permissions migration" 2024-09-06T14:25:36.665 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.53465078Z level=info msg="Migration successfully executed" id="managed permissions migration" duration=331.261µs 2024-09-06T14:25:36.665 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.535199668Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-09-06T14:25:36.665 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.535331393Z level=info msg="Migration successfully executed" id="managed folder permissions alert actions migration" duration=131.669µs 2024-09-06T14:25:36.665 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.536057339Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-09-06T14:25:36.665 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.536910145Z level=info msg="Migration successfully executed" id="RBAC action name migrator" duration=852.997µs 2024-09-06T14:25:36.665 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.537348131Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-09-06T14:25:36.665 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.542211514Z level=info msg="Migration successfully executed" id="Add UID column to playlist" duration=4.863158ms 2024-09-06T14:25:36.666 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.542579166Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-09-06T14:25:36.666 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.542686691Z level=info msg="Migration successfully executed" id="Update uid column values in playlist" duration=107.745µs 2024-09-06T14:25:36.666 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.543453008Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-09-06T14:25:36.666 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.54423241Z level=info msg="Migration successfully executed" id="Add index for uid in playlist" duration=779.543µs 2024-09-06T14:25:36.666 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.54620903Z level=info msg="Executing migration" id="update group index for alert rules" 2024-09-06T14:25:36.666 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.54645774Z level=info msg="Migration successfully executed" id="update group index for alert rules" duration=248.532µs 2024-09-06T14:25:36.666 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.549256537Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-09-06T14:25:36.666 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.549550726Z level=info msg="Migration successfully executed" id="managed folder permissions alert actions repeated migration" duration=296.429µs 2024-09-06T14:25:36.666 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.552169127Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-09-06T14:25:36.666 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.55262886Z level=info msg="Migration successfully executed" id="admin only folder/dashboard permission" duration=459.576µs 2024-09-06T14:25:36.666 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.554962857Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-09-06T14:25:36.666 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.562909128Z level=info msg="Migration successfully executed" id="add action column to seed_assignment" duration=7.942179ms 2024-09-06T14:25:36.666 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.563644855Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-09-06T14:25:36.666 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.568791004Z level=info msg="Migration successfully executed" id="add scope column to seed_assignment" duration=5.148171ms 2024-09-06T14:25:36.667 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.569329119Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-09-06T14:25:36.667 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.570119343Z level=info msg="Migration successfully executed" id="remove unique index builtin_role_role_name before nullable update" duration=790.318µs 2024-09-06T14:25:36.667 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.578120471Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-09-06T14:25:36.667 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.636551479Z level=info msg="Migration successfully executed" id="update seed_assignment role_name column to nullable" duration=58.428297ms 2024-09-06T14:25:36.667 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.637054864Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-09-06T14:25:36.667 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.637853773Z level=info msg="Migration successfully executed" id="add unique index builtin_role_name back" duration=799.563µs 2024-09-06T14:25:36.667 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.638215789Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-09-06T14:25:36.667 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.638950355Z level=info msg="Migration successfully executed" id="add unique index builtin_role_action_scope" duration=732.181µs 2024-09-06T14:25:36.667 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.639308557Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-09-06T14:25:36.667 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.656065405Z level=info msg="Migration successfully executed" id="add primary key to seed_assigment" duration=16.755ms 2024-09-06T14:25:36.667 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.656598436Z level=info msg="Executing migration" id="add origin column to seed_assignment" 2024-09-06T14:25:36.797 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:36 smithi142 ceph-mon[37372]: pgmap v23: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:25:36.797 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:36 smithi142 ceph-mon[37372]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:36.797 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:36 smithi142 ceph-mon[37372]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:36.797 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:36 smithi142 ceph-mon[37372]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:36.797 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:36 smithi142 ceph-mon[37372]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:36.797 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:36 smithi142 ceph-mon[37372]: Deploying daemon node-exporter.a on smithi142 2024-09-06T14:25:36.797 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:36 smithi142 ceph-mon[37372]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:36.797 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:36 smithi142 ceph-mon[29958]: pgmap v23: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:25:36.797 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:36 smithi142 ceph-mon[29958]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:36.797 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:36 smithi142 ceph-mon[29958]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:36.798 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:36 smithi142 ceph-mon[29958]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:36.798 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:36 smithi142 ceph-mon[29958]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:36.798 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:36 smithi142 ceph-mon[29958]: Deploying daemon node-exporter.a on smithi142 2024-09-06T14:25:36.798 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:36 smithi142 ceph-mon[29958]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:36.799 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:36.802+0000 7fbf6d90f640 1 -- 172.21.15.142:0/2289133884 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fbf68104870 con 0x7fbf681019c0 2024-09-06T14:25:36.799 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:36.803+0000 7fbf6c90d640 1 -- 172.21.15.142:0/2289133884 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+212 (secure 0 0 0) 0x7fbf5806b290 con 0x7fbf681019c0 2024-09-06T14:25:36.800 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:25:36.800 INFO:teuthology.orchestra.run.smithi142.stdout:{"status":"HEALTH_WARN","checks":{"MGR_MODULE_DEPENDENCY":{"severity":"HEALTH_WARN","summary":{"message":"Module 'volumes' has failed dependency: No module named 'ceph.fs'","count":1},"muted":false}},"mutes":[]} 2024-09-06T14:25:36.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:36.805+0000 7fbf6d90f640 1 -- 172.21.15.142:0/2289133884 >> v2:172.21.15.142:6800/2944545612 conn(0x7fbf50077620 msgr2=0x7fbf50079ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:25:36.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:36.805+0000 7fbf6d90f640 1 --2- 172.21.15.142:0/2289133884 >> v2:172.21.15.142:6800/2944545612 conn(0x7fbf50077620 0x7fbf50079ae0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fbf5c031440 tx=0x7fbf5c031850 comp rx=0 tx=0).stop 2024-09-06T14:25:36.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:36.805+0000 7fbf6d90f640 1 -- 172.21.15.142:0/2289133884 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbf681019c0 msgr2=0x7fbf6819a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:25:36.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:36.805+0000 7fbf6d90f640 1 --2- 172.21.15.142:0/2289133884 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbf681019c0 0x7fbf6819a920 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fbf5800e9f0 tx=0x7fbf5800edf0 comp rx=0 tx=0).stop 2024-09-06T14:25:36.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:36.805+0000 7fbf6d90f640 1 -- 172.21.15.142:0/2289133884 shutdown_connections 2024-09-06T14:25:36.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:36.805+0000 7fbf6d90f640 1 --2- 172.21.15.142:0/2289133884 >> v2:172.21.15.142:6800/2944545612 conn(0x7fbf50077620 0x7fbf50079ae0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:36.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:36.805+0000 7fbf6d90f640 1 --2- 172.21.15.142:0/2289133884 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbf68107d50 0x7fbf681a0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:36.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:36.805+0000 7fbf6d90f640 1 --2- 172.21.15.142:0/2289133884 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbf681073d0 0x7fbf6819ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:36.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:36.805+0000 7fbf6d90f640 1 --2- 172.21.15.142:0/2289133884 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbf681019c0 0x7fbf6819a920 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:36.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:36.805+0000 7fbf6d90f640 1 -- 172.21.15.142:0/2289133884 >> 172.21.15.142:0/2289133884 conn(0x7fbf680fd430 msgr2=0x7fbf6810f760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:25:36.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:36.805+0000 7fbf6d90f640 1 -- 172.21.15.142:0/2289133884 shutdown_connections 2024-09-06T14:25:36.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:36.805+0000 7fbf6d90f640 1 -- 172.21.15.142:0/2289133884 wait complete. 2024-09-06T14:25:36.910 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.662086949Z level=info msg="Migration successfully executed" id="add origin column to seed_assignment" duration=5.492833ms 2024-09-06T14:25:36.910 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.662515222Z level=info msg="Executing migration" id="add origin to plugin seed_assignment" 2024-09-06T14:25:36.910 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.6627484Z level=info msg="Migration successfully executed" id="add origin to plugin seed_assignment" duration=231.435µs 2024-09-06T14:25:36.910 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.663327388Z level=info msg="Executing migration" id="prevent seeding OnCall access" 2024-09-06T14:25:36.911 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.66345874Z level=info msg="Migration successfully executed" id="prevent seeding OnCall access" duration=131.377µs 2024-09-06T14:25:36.911 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.664167578Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-09-06T14:25:36.911 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.66430383Z level=info msg="Migration successfully executed" id="managed folder permissions alert actions repeated fixed migration" duration=137.702µs 2024-09-06T14:25:36.911 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.664706419Z level=info msg="Executing migration" id="managed folder permissions library panel actions migration" 2024-09-06T14:25:36.911 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.664837189Z level=info msg="Migration successfully executed" id="managed folder permissions library panel actions migration" duration=130.842µs 2024-09-06T14:25:36.911 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.665525304Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-09-06T14:25:36.911 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.665665614Z level=info msg="Migration successfully executed" id="migrate external alertmanagers to datsourcse" duration=140.297µs 2024-09-06T14:25:36.911 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.666378895Z level=info msg="Executing migration" id="create folder table" 2024-09-06T14:25:36.911 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.666993525Z level=info msg="Migration successfully executed" id="create folder table" duration=614.446µs 2024-09-06T14:25:36.911 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.667346861Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-09-06T14:25:36.911 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.668170006Z level=info msg="Migration successfully executed" id="Add index for parent_uid" duration=822.63µs 2024-09-06T14:25:36.911 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.668573888Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-09-06T14:25:36.911 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.669352043Z level=info msg="Migration successfully executed" id="Add unique index for folder.uid and folder.org_id" duration=778.129µs 2024-09-06T14:25:36.912 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.669741016Z level=info msg="Executing migration" id="Update folder title length" 2024-09-06T14:25:36.912 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.669761642Z level=info msg="Migration successfully executed" id="Update folder title length" duration=20.834µs 2024-09-06T14:25:36.912 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.670575667Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-09-06T14:25:36.912 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.671315776Z level=info msg="Migration successfully executed" id="Add unique index for folder.title and folder.parent_uid" duration=739.81µs 2024-09-06T14:25:36.912 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.671699086Z level=info msg="Executing migration" id="Remove unique index for folder.title and folder.parent_uid" 2024-09-06T14:25:36.912 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.672394941Z level=info msg="Migration successfully executed" id="Remove unique index for folder.title and folder.parent_uid" duration=694.057µs 2024-09-06T14:25:36.912 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.672801615Z level=info msg="Executing migration" id="Add unique index for title, parent_uid, and org_id" 2024-09-06T14:25:36.912 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.673554505Z level=info msg="Migration successfully executed" id="Add unique index for title, parent_uid, and org_id" duration=752.594µs 2024-09-06T14:25:36.912 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.674016955Z level=info msg="Executing migration" id="Sync dashboard and folder table" 2024-09-06T14:25:36.912 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.674337641Z level=info msg="Migration successfully executed" id="Sync dashboard and folder table" duration=320.522µs 2024-09-06T14:25:36.912 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.67485722Z level=info msg="Executing migration" id="Remove ghost folders from the folder table" 2024-09-06T14:25:36.912 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.675075633Z level=info msg="Migration successfully executed" id="Remove ghost folders from the folder table" duration=218.341µs 2024-09-06T14:25:36.912 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.675689505Z level=info msg="Executing migration" id="Remove unique index UQE_folder_uid_org_id" 2024-09-06T14:25:36.913 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.676382919Z level=info msg="Migration successfully executed" id="Remove unique index UQE_folder_uid_org_id" duration=693.356µs 2024-09-06T14:25:36.913 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.676768523Z level=info msg="Executing migration" id="Add unique index UQE_folder_org_id_uid" 2024-09-06T14:25:36.913 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.677495646Z level=info msg="Migration successfully executed" id="Add unique index UQE_folder_org_id_uid" duration=728.32µs 2024-09-06T14:25:36.913 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.677856511Z level=info msg="Executing migration" id="Remove unique index UQE_folder_title_parent_uid_org_id" 2024-09-06T14:25:36.913 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.678539091Z level=info msg="Migration successfully executed" id="Remove unique index UQE_folder_title_parent_uid_org_id" duration=682.49µs 2024-09-06T14:25:36.913 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.678898472Z level=info msg="Executing migration" id="Add unique index UQE_folder_org_id_parent_uid_title" 2024-09-06T14:25:36.913 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.679652831Z level=info msg="Migration successfully executed" id="Add unique index UQE_folder_org_id_parent_uid_title" duration=752.24µs 2024-09-06T14:25:36.913 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.680014401Z level=info msg="Executing migration" id="Remove index IDX_folder_parent_uid_org_id" 2024-09-06T14:25:36.913 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.680731011Z level=info msg="Migration successfully executed" id="Remove index IDX_folder_parent_uid_org_id" duration=716.396µs 2024-09-06T14:25:36.913 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.681089849Z level=info msg="Executing migration" id="create anon_device table" 2024-09-06T14:25:36.913 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.681692399Z level=info msg="Migration successfully executed" id="create anon_device table" duration=602.347µs 2024-09-06T14:25:36.913 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.68207745Z level=info msg="Executing migration" id="add unique index anon_device.device_id" 2024-09-06T14:25:36.913 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.682939763Z level=info msg="Migration successfully executed" id="add unique index anon_device.device_id" duration=862.119µs 2024-09-06T14:25:36.914 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.683331606Z level=info msg="Executing migration" id="add index anon_device.updated_at" 2024-09-06T14:25:36.914 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.684073163Z level=info msg="Migration successfully executed" id="add index anon_device.updated_at" duration=741.369µs 2024-09-06T14:25:36.914 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.684484796Z level=info msg="Executing migration" id="create signing_key table" 2024-09-06T14:25:36.914 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.685135843Z level=info msg="Migration successfully executed" id="create signing_key table" duration=650.912µs 2024-09-06T14:25:36.914 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.685583599Z level=info msg="Executing migration" id="add unique index signing_key.key_id" 2024-09-06T14:25:36.914 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.686295334Z level=info msg="Migration successfully executed" id="add unique index signing_key.key_id" duration=712.01µs 2024-09-06T14:25:36.914 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.686641473Z level=info msg="Executing migration" id="set legacy alert migration status in kvstore" 2024-09-06T14:25:36.914 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.687329767Z level=info msg="Migration successfully executed" id="set legacy alert migration status in kvstore" duration=688.187µs 2024-09-06T14:25:36.914 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.687669648Z level=info msg="Executing migration" id="migrate record of created folders during legacy migration to kvstore" 2024-09-06T14:25:36.915 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.687837512Z level=info msg="Migration successfully executed" id="migrate record of created folders during legacy migration to kvstore" duration=168.315µs 2024-09-06T14:25:36.915 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.688558033Z level=info msg="Executing migration" id="Add folder_uid for dashboard" 2024-09-06T14:25:36.915 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.694229783Z level=info msg="Migration successfully executed" id="Add folder_uid for dashboard" duration=5.671464ms 2024-09-06T14:25:36.915 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.694579139Z level=info msg="Executing migration" id="Populate dashboard folder_uid column" 2024-09-06T14:25:36.915 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.69506138Z level=info msg="Migration successfully executed" id="Populate dashboard folder_uid column" duration=483.052µs 2024-09-06T14:25:36.915 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.697446923Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_folder_uid_title" 2024-09-06T14:25:36.915 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.698180509Z level=info msg="Migration successfully executed" id="Add unique index for dashboard_org_id_folder_uid_title" duration=733.281µs 2024-09-06T14:25:36.915 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.700354466Z level=info msg="Executing migration" id="Delete unique index for dashboard_org_id_folder_id_title" 2024-09-06T14:25:36.916 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.70110064Z level=info msg="Migration successfully executed" id="Delete unique index for dashboard_org_id_folder_id_title" duration=744.444µs 2024-09-06T14:25:36.916 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.703151401Z level=info msg="Executing migration" id="Delete unique index for dashboard_org_id_folder_uid_title" 2024-09-06T14:25:36.916 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.703880214Z level=info msg="Migration successfully executed" id="Delete unique index for dashboard_org_id_folder_uid_title" duration=728.624µs 2024-09-06T14:25:36.916 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.705878791Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_folder_uid_title_is_folder" 2024-09-06T14:25:36.916 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.706633043Z level=info msg="Migration successfully executed" id="Add unique index for dashboard_org_id_folder_uid_title_is_folder" duration=754.479µs 2024-09-06T14:25:36.916 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.708641024Z level=info msg="Executing migration" id="Restore index for dashboard_org_id_folder_id_title" 2024-09-06T14:25:36.916 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.709383946Z level=info msg="Migration successfully executed" id="Restore index for dashboard_org_id_folder_id_title" duration=742.559µs 2024-09-06T14:25:36.916 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.71154288Z level=info msg="Executing migration" id="create sso_setting table" 2024-09-06T14:25:36.916 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.712237175Z level=info msg="Migration successfully executed" id="create sso_setting table" duration=694.017µs 2024-09-06T14:25:36.916 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.714266765Z level=info msg="Executing migration" id="copy kvstore migration status to each org" 2024-09-06T14:25:36.916 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.714753255Z level=info msg="Migration successfully executed" id="copy kvstore migration status to each org" duration=486.87µs 2024-09-06T14:25:36.916 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.71509251Z level=info msg="Executing migration" id="add back entry for orgid=0 migrated status" 2024-09-06T14:25:36.916 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.715266365Z level=info msg="Migration successfully executed" id="add back entry for orgid=0 migrated status" duration=174.465µs 2024-09-06T14:25:36.916 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.717837172Z level=info msg="Executing migration" id="alter kv_store.value to longtext" 2024-09-06T14:25:36.917 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.717890866Z level=info msg="Migration successfully executed" id="alter kv_store.value to longtext" duration=53.657µs 2024-09-06T14:25:36.917 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.72050737Z level=info msg="Executing migration" id="add notification_settings column to alert_rule table" 2024-09-06T14:25:36.917 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.72572359Z level=info msg="Migration successfully executed" id="add notification_settings column to alert_rule table" duration=5.215712ms 2024-09-06T14:25:36.917 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.726120866Z level=info msg="Executing migration" id="add notification_settings column to alert_rule_version table" 2024-09-06T14:25:36.917 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.731449231Z level=info msg="Migration successfully executed" id="add notification_settings column to alert_rule_version table" duration=5.327906ms 2024-09-06T14:25:36.917 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.731846962Z level=info msg="Executing migration" id="removing scope from alert.instances:read action migration" 2024-09-06T14:25:36.917 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.732074903Z level=info msg="Migration successfully executed" id="removing scope from alert.instances:read action migration" duration=227.796µs 2024-09-06T14:25:36.917 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=migrator t=2024-09-06T14:25:36.73468919Z level=info msg="migrations completed" performed=547 skipped=0 duration=1.223712942s 2024-09-06T14:25:36.917 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=sqlstore t=2024-09-06T14:25:36.735582016Z level=info msg="Created default organization" 2024-09-06T14:25:36.917 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=secrets t=2024-09-06T14:25:36.737835899Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-09-06T14:25:36.917 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=plugin.store t=2024-09-06T14:25:36.755758394Z level=info msg="Loading plugins..." 2024-09-06T14:25:36.917 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=local.finder t=2024-09-06T14:25:36.797800185Z level=warn msg="Skipping finding plugins as directory does not exist" path=/usr/share/grafana/plugins-bundled 2024-09-06T14:25:36.917 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=plugin.store t=2024-09-06T14:25:36.797818781Z level=info msg="Plugins loaded" count=55 duration=42.061184ms 2024-09-06T14:25:36.918 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=query_data t=2024-09-06T14:25:36.79997462Z level=info msg="Query Service initialization" 2024-09-06T14:25:36.918 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=live.push_http t=2024-09-06T14:25:36.802376989Z level=info msg="Live Push Gateway initialization" 2024-09-06T14:25:36.918 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=ngalert.migration t=2024-09-06T14:25:36.803411028Z level=info msg=Starting 2024-09-06T14:25:36.918 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=ngalert.migration t=2024-09-06T14:25:36.803689991Z level=info msg="Applying transition" currentType=Legacy desiredType=UnifiedAlerting cleanOnDowngrade=false cleanOnUpgrade=false 2024-09-06T14:25:36.918 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=ngalert.migration orgID=1 t=2024-09-06T14:25:36.803947526Z level=info msg="Migrating alerts for organisation" 2024-09-06T14:25:36.918 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=ngalert.migration orgID=1 t=2024-09-06T14:25:36.80436467Z level=info msg="Alerts found to migrate" alerts=0 2024-09-06T14:25:36.918 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=ngalert.migration t=2024-09-06T14:25:36.805495421Z level=info msg="Completed alerting migration" 2024-09-06T14:25:36.918 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=ngalert.state.manager t=2024-09-06T14:25:36.819287098Z level=info msg="Running in alternative execution of Error/NoData mode" 2024-09-06T14:25:36.918 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=infra.usagestats.collector t=2024-09-06T14:25:36.820709354Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-09-06T14:25:36.918 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=provisioning.datasources t=2024-09-06T14:25:36.822376514Z level=info msg="inserting datasource from configuration" name=Dashboard1 uid=P43CA22E17D0F9596 2024-09-06T14:25:36.918 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=provisioning.datasources t=2024-09-06T14:25:36.835334728Z level=info msg="inserting datasource from configuration" name=Loki uid=P8E80F9AEF21F6940 2024-09-06T14:25:36.918 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=provisioning.alerting t=2024-09-06T14:25:36.848397622Z level=info msg="starting to provision alerting" 2024-09-06T14:25:36.918 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=provisioning.alerting t=2024-09-06T14:25:36.848412799Z level=info msg="finished to provision alerting" 2024-09-06T14:25:36.918 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=ngalert.state.manager t=2024-09-06T14:25:36.848532258Z level=info msg="Warming state cache for startup" 2024-09-06T14:25:36.919 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=ngalert.state.manager t=2024-09-06T14:25:36.848901767Z level=info msg="State cache has been initialized" states=0 duration=368.895µs 2024-09-06T14:25:36.919 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=ngalert.multiorg.alertmanager t=2024-09-06T14:25:36.848577469Z level=info msg="Starting MultiOrg Alertmanager" 2024-09-06T14:25:36.919 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=ngalert.scheduler t=2024-09-06T14:25:36.849167993Z level=info msg="Starting scheduler" tickInterval=10s maxAttempts=1 2024-09-06T14:25:36.919 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=grafanaStorageLogger t=2024-09-06T14:25:36.849193269Z level=info msg="Storage starting" 2024-09-06T14:25:36.919 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=ticker t=2024-09-06T14:25:36.849214633Z level=info msg=starting first_tick=2024-09-06T14:25:40Z 2024-09-06T14:25:36.919 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=http.server t=2024-09-06T14:25:36.85102429Z level=info msg="HTTP Server TLS settings" MinTLSVersion=TLS1.2 configuredciphers=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA 2024-09-06T14:25:36.919 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=http.server t=2024-09-06T14:25:36.851269836Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-09-06T14:25:36.919 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=provisioning.dashboard t=2024-09-06T14:25:36.857612136Z level=info msg="starting to provision dashboards" 2024-09-06T14:25:36.919 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=sqlstore.transactions t=2024-09-06T14:25:36.891048868Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-06T14:25:36.919 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=sqlstore.transactions t=2024-09-06T14:25:36.902399636Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-06T14:25:36.919 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=sqlstore.transactions t=2024-09-06T14:25:36.913507092Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=1 code="database is locked" 2024-09-06T14:25:37.297 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 06 14:25:37 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-iscsi-iscsi-a[59853]: debug there is no tcmu-runner data available 2024-09-06T14:25:37.297 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=plugins.update.checker t=2024-09-06T14:25:36.923502778Z level=info msg="Update check succeeded" duration=74.667074ms 2024-09-06T14:25:37.297 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=sqlstore.transactions t=2024-09-06T14:25:36.924488297Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=2 code="database is locked" 2024-09-06T14:25:37.297 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=grafana.update.checker t=2024-09-06T14:25:36.92870342Z level=info msg="Update check succeeded" duration=80.067778ms 2024-09-06T14:25:37.297 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=sqlstore.transactions t=2024-09-06T14:25:36.935657286Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=3 code="database is locked" 2024-09-06T14:25:37.297 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=sqlstore.transactions t=2024-09-06T14:25:36.944784338Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-06T14:25:37.297 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:36 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=sqlstore.transactions t=2024-09-06T14:25:36.946923267Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=4 code="database is locked" 2024-09-06T14:25:37.298 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:37 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=provisioning.dashboard t=2024-09-06T14:25:37.051412627Z level=info msg="finished to provision dashboards" 2024-09-06T14:25:37.797 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:37 smithi162 ceph-mon[33787]: from='client.? 172.21.15.142:0/2289133884' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:25:37.797 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:37 smithi162 ceph-mon[33787]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-06T14:25:37.797 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:37 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=grafana-apiserver t=2024-09-06T14:25:37.405094417Z level=info msg="Adding GroupVersion playlist.grafana.app v0alpha1 to ResourceManager" 2024-09-06T14:25:37.797 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:25:37 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=grafana-apiserver t=2024-09-06T14:25:37.405465031Z level=info msg="Adding GroupVersion featuretoggle.grafana.app v0alpha1 to ResourceManager" 2024-09-06T14:25:37.958 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:37 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/2289133884' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:25:37.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:37 smithi142 ceph-mon[29958]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-06T14:25:37.959 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:37 smithi142 ceph-mon[37372]: from='client.? 172.21.15.142:0/2289133884' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:25:37.959 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:37 smithi142 ceph-mon[37372]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-06T14:25:38.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:38 smithi142 ceph-mon[29958]: pgmap v24: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:25:38.959 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:38 smithi142 ceph-mon[37372]: pgmap v24: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:25:38.960 INFO:journalctl@ceph.node-exporter.a.smithi142.stdout:Sep 06 14:25:38 smithi142 bash[71392]: Writing manifest to image destination 2024-09-06T14:25:39.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:38 smithi162 ceph-mon[33787]: pgmap v24: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:25:39.406 INFO:journalctl@ceph.node-exporter.a.smithi142.stdout:Sep 06 14:25:39 smithi142 podman[71392]: 2024-09-06 14:25:39.050598629 +0000 UTC m=+2.346785504 image pull 72c9c208898624938c9e4183d6686ea4a5fd3f912bc29bc3f00147924c521a3e quay.io/prometheus/node-exporter:v1.7.0 2024-09-06T14:25:39.406 INFO:journalctl@ceph.node-exporter.a.smithi142.stdout:Sep 06 14:25:39 smithi142 podman[71392]: 2024-09-06 14:25:39.157559832 +0000 UTC m=+2.453746750 container create ffc6b152297c8bf3e77095156e3b4daf087ef3537f23553b3dd13ca97a39f587 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-06T14:25:39.709 INFO:journalctl@ceph.node-exporter.a.smithi142.stdout:Sep 06 14:25:39 smithi142 podman[71392]: 2024-09-06 14:25:39.410429682 +0000 UTC m=+2.706616554 container init ffc6b152297c8bf3e77095156e3b4daf087ef3537f23553b3dd13ca97a39f587 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-06T14:25:39.709 INFO:journalctl@ceph.node-exporter.a.smithi142.stdout:Sep 06 14:25:39 smithi142 podman[71392]: 2024-09-06 14:25:39.413521198 +0000 UTC m=+2.709708080 container start ffc6b152297c8bf3e77095156e3b4daf087ef3537f23553b3dd13ca97a39f587 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-06T14:25:39.709 INFO:journalctl@ceph.node-exporter.a.smithi142.stdout:Sep 06 14:25:39 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-a[71484]: ts=2024-09-06T14:25:39.417Z caller=node_exporter.go:192 level=info msg="Starting node_exporter" version="(version=1.7.0, branch=HEAD, revision=7333465abf9efba81876303bb57e6fadb946041b)" 2024-09-06T14:25:39.709 INFO:journalctl@ceph.node-exporter.a.smithi142.stdout:Sep 06 14:25:39 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-a[71484]: ts=2024-09-06T14:25:39.417Z caller=node_exporter.go:193 level=info msg="Build context" build_context="(go=go1.21.4, platform=linux/amd64, user=root@35918982f6d8, date=20231112-23:53:35, tags=netgo osusergo static_build)" 2024-09-06T14:25:39.710 INFO:journalctl@ceph.node-exporter.a.smithi142.stdout:Sep 06 14:25:39 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-a[71484]: ts=2024-09-06T14:25:39.418Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-09-06T14:25:39.710 INFO:journalctl@ceph.node-exporter.a.smithi142.stdout:Sep 06 14:25:39 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-a[71484]: ts=2024-09-06T14:25:39.418Z caller=diskstats_linux.go:265 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-09-06T14:25:39.710 INFO:journalctl@ceph.node-exporter.a.smithi142.stdout:Sep 06 14:25:39 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-a[71484]: ts=2024-09-06T14:25:39.418Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-09-06T14:25:39.710 INFO:journalctl@ceph.node-exporter.a.smithi142.stdout:Sep 06 14:25:39 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-a[71484]: ts=2024-09-06T14:25:39.418Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-09-06T14:25:39.710 INFO:journalctl@ceph.node-exporter.a.smithi142.stdout:Sep 06 14:25:39 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-a[71484]: ts=2024-09-06T14:25:39.418Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-09-06T14:25:39.710 INFO:journalctl@ceph.node-exporter.a.smithi142.stdout:Sep 06 14:25:39 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-a[71484]: ts=2024-09-06T14:25:39.418Z caller=node_exporter.go:117 level=info collector=arp 2024-09-06T14:25:39.710 INFO:journalctl@ceph.node-exporter.a.smithi142.stdout:Sep 06 14:25:39 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-a[71484]: ts=2024-09-06T14:25:39.418Z caller=node_exporter.go:117 level=info collector=bcache 2024-09-06T14:25:39.710 INFO:journalctl@ceph.node-exporter.a.smithi142.stdout:Sep 06 14:25:39 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-a[71484]: ts=2024-09-06T14:25:39.418Z caller=node_exporter.go:117 level=info collector=bonding 2024-09-06T14:25:39.710 INFO:journalctl@ceph.node-exporter.a.smithi142.stdout:Sep 06 14:25:39 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-a[71484]: ts=2024-09-06T14:25:39.418Z caller=node_exporter.go:117 level=info collector=btrfs 2024-09-06T14:25:39.710 INFO:journalctl@ceph.node-exporter.a.smithi142.stdout:Sep 06 14:25:39 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-a[71484]: ts=2024-09-06T14:25:39.418Z caller=node_exporter.go:117 level=info collector=conntrack 2024-09-06T14:25:39.710 INFO:journalctl@ceph.node-exporter.a.smithi142.stdout:Sep 06 14:25:39 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-a[71484]: ts=2024-09-06T14:25:39.418Z caller=node_exporter.go:117 level=info collector=cpu 2024-09-06T14:25:39.710 INFO:journalctl@ceph.node-exporter.a.smithi142.stdout:Sep 06 14:25:39 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-a[71484]: ts=2024-09-06T14:25:39.418Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-09-06T14:25:39.711 INFO:journalctl@ceph.node-exporter.a.smithi142.stdout:Sep 06 14:25:39 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-a[71484]: ts=2024-09-06T14:25:39.418Z caller=node_exporter.go:117 level=info collector=diskstats 2024-09-06T14:25:39.711 INFO:journalctl@ceph.node-exporter.a.smithi142.stdout:Sep 06 14:25:39 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-a[71484]: ts=2024-09-06T14:25:39.418Z caller=node_exporter.go:117 level=info collector=dmi 2024-09-06T14:25:39.711 INFO:journalctl@ceph.node-exporter.a.smithi142.stdout:Sep 06 14:25:39 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-a[71484]: ts=2024-09-06T14:25:39.418Z caller=node_exporter.go:117 level=info collector=edac 2024-09-06T14:25:39.711 INFO:journalctl@ceph.node-exporter.a.smithi142.stdout:Sep 06 14:25:39 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-a[71484]: ts=2024-09-06T14:25:39.418Z caller=node_exporter.go:117 level=info collector=entropy 2024-09-06T14:25:39.711 INFO:journalctl@ceph.node-exporter.a.smithi142.stdout:Sep 06 14:25:39 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-a[71484]: ts=2024-09-06T14:25:39.418Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-09-06T14:25:39.711 INFO:journalctl@ceph.node-exporter.a.smithi142.stdout:Sep 06 14:25:39 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-a[71484]: ts=2024-09-06T14:25:39.418Z caller=node_exporter.go:117 level=info collector=filefd 2024-09-06T14:25:39.711 INFO:journalctl@ceph.node-exporter.a.smithi142.stdout:Sep 06 14:25:39 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-a[71484]: ts=2024-09-06T14:25:39.419Z caller=node_exporter.go:117 level=info collector=filesystem 2024-09-06T14:25:39.712 INFO:journalctl@ceph.node-exporter.a.smithi142.stdout:Sep 06 14:25:39 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-a[71484]: ts=2024-09-06T14:25:39.419Z caller=node_exporter.go:117 level=info collector=hwmon 2024-09-06T14:25:39.712 INFO:journalctl@ceph.node-exporter.a.smithi142.stdout:Sep 06 14:25:39 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-a[71484]: ts=2024-09-06T14:25:39.419Z caller=node_exporter.go:117 level=info collector=infiniband 2024-09-06T14:25:39.712 INFO:journalctl@ceph.node-exporter.a.smithi142.stdout:Sep 06 14:25:39 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-a[71484]: ts=2024-09-06T14:25:39.419Z caller=node_exporter.go:117 level=info collector=ipvs 2024-09-06T14:25:39.712 INFO:journalctl@ceph.node-exporter.a.smithi142.stdout:Sep 06 14:25:39 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-a[71484]: ts=2024-09-06T14:25:39.419Z caller=node_exporter.go:117 level=info collector=loadavg 2024-09-06T14:25:39.712 INFO:journalctl@ceph.node-exporter.a.smithi142.stdout:Sep 06 14:25:39 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-a[71484]: ts=2024-09-06T14:25:39.419Z caller=node_exporter.go:117 level=info collector=mdadm 2024-09-06T14:25:39.712 INFO:journalctl@ceph.node-exporter.a.smithi142.stdout:Sep 06 14:25:39 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-a[71484]: ts=2024-09-06T14:25:39.419Z caller=node_exporter.go:117 level=info collector=meminfo 2024-09-06T14:25:39.712 INFO:journalctl@ceph.node-exporter.a.smithi142.stdout:Sep 06 14:25:39 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-a[71484]: ts=2024-09-06T14:25:39.419Z caller=node_exporter.go:117 level=info collector=netclass 2024-09-06T14:25:39.712 INFO:journalctl@ceph.node-exporter.a.smithi142.stdout:Sep 06 14:25:39 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-a[71484]: ts=2024-09-06T14:25:39.419Z caller=node_exporter.go:117 level=info collector=netdev 2024-09-06T14:25:39.712 INFO:journalctl@ceph.node-exporter.a.smithi142.stdout:Sep 06 14:25:39 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-a[71484]: ts=2024-09-06T14:25:39.419Z caller=node_exporter.go:117 level=info collector=netstat 2024-09-06T14:25:39.712 INFO:journalctl@ceph.node-exporter.a.smithi142.stdout:Sep 06 14:25:39 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-a[71484]: ts=2024-09-06T14:25:39.419Z caller=node_exporter.go:117 level=info collector=nfs 2024-09-06T14:25:39.712 INFO:journalctl@ceph.node-exporter.a.smithi142.stdout:Sep 06 14:25:39 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-a[71484]: ts=2024-09-06T14:25:39.419Z caller=node_exporter.go:117 level=info collector=nfsd 2024-09-06T14:25:39.712 INFO:journalctl@ceph.node-exporter.a.smithi142.stdout:Sep 06 14:25:39 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-a[71484]: ts=2024-09-06T14:25:39.419Z caller=node_exporter.go:117 level=info collector=nvme 2024-09-06T14:25:39.713 INFO:journalctl@ceph.node-exporter.a.smithi142.stdout:Sep 06 14:25:39 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-a[71484]: ts=2024-09-06T14:25:39.419Z caller=node_exporter.go:117 level=info collector=os 2024-09-06T14:25:39.713 INFO:journalctl@ceph.node-exporter.a.smithi142.stdout:Sep 06 14:25:39 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-a[71484]: ts=2024-09-06T14:25:39.419Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-09-06T14:25:39.713 INFO:journalctl@ceph.node-exporter.a.smithi142.stdout:Sep 06 14:25:39 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-a[71484]: ts=2024-09-06T14:25:39.419Z caller=node_exporter.go:117 level=info collector=pressure 2024-09-06T14:25:39.713 INFO:journalctl@ceph.node-exporter.a.smithi142.stdout:Sep 06 14:25:39 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-a[71484]: ts=2024-09-06T14:25:39.419Z caller=node_exporter.go:117 level=info collector=rapl 2024-09-06T14:25:39.713 INFO:journalctl@ceph.node-exporter.a.smithi142.stdout:Sep 06 14:25:39 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-a[71484]: ts=2024-09-06T14:25:39.419Z caller=node_exporter.go:117 level=info collector=schedstat 2024-09-06T14:25:39.713 INFO:journalctl@ceph.node-exporter.a.smithi142.stdout:Sep 06 14:25:39 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-a[71484]: ts=2024-09-06T14:25:39.419Z caller=node_exporter.go:117 level=info collector=selinux 2024-09-06T14:25:39.713 INFO:journalctl@ceph.node-exporter.a.smithi142.stdout:Sep 06 14:25:39 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-a[71484]: ts=2024-09-06T14:25:39.419Z caller=node_exporter.go:117 level=info collector=sockstat 2024-09-06T14:25:39.713 INFO:journalctl@ceph.node-exporter.a.smithi142.stdout:Sep 06 14:25:39 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-a[71484]: ts=2024-09-06T14:25:39.419Z caller=node_exporter.go:117 level=info collector=softnet 2024-09-06T14:25:39.713 INFO:journalctl@ceph.node-exporter.a.smithi142.stdout:Sep 06 14:25:39 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-a[71484]: ts=2024-09-06T14:25:39.419Z caller=node_exporter.go:117 level=info collector=stat 2024-09-06T14:25:39.713 INFO:journalctl@ceph.node-exporter.a.smithi142.stdout:Sep 06 14:25:39 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-a[71484]: ts=2024-09-06T14:25:39.419Z caller=node_exporter.go:117 level=info collector=tapestats 2024-09-06T14:25:39.713 INFO:journalctl@ceph.node-exporter.a.smithi142.stdout:Sep 06 14:25:39 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-a[71484]: ts=2024-09-06T14:25:39.419Z caller=node_exporter.go:117 level=info collector=textfile 2024-09-06T14:25:39.713 INFO:journalctl@ceph.node-exporter.a.smithi142.stdout:Sep 06 14:25:39 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-a[71484]: ts=2024-09-06T14:25:39.419Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-09-06T14:25:39.713 INFO:journalctl@ceph.node-exporter.a.smithi142.stdout:Sep 06 14:25:39 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-a[71484]: ts=2024-09-06T14:25:39.419Z caller=node_exporter.go:117 level=info collector=time 2024-09-06T14:25:39.713 INFO:journalctl@ceph.node-exporter.a.smithi142.stdout:Sep 06 14:25:39 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-a[71484]: ts=2024-09-06T14:25:39.419Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-09-06T14:25:39.714 INFO:journalctl@ceph.node-exporter.a.smithi142.stdout:Sep 06 14:25:39 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-a[71484]: ts=2024-09-06T14:25:39.419Z caller=node_exporter.go:117 level=info collector=uname 2024-09-06T14:25:39.714 INFO:journalctl@ceph.node-exporter.a.smithi142.stdout:Sep 06 14:25:39 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-a[71484]: ts=2024-09-06T14:25:39.419Z caller=node_exporter.go:117 level=info collector=vmstat 2024-09-06T14:25:39.714 INFO:journalctl@ceph.node-exporter.a.smithi142.stdout:Sep 06 14:25:39 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-a[71484]: ts=2024-09-06T14:25:39.419Z caller=node_exporter.go:117 level=info collector=xfs 2024-09-06T14:25:39.714 INFO:journalctl@ceph.node-exporter.a.smithi142.stdout:Sep 06 14:25:39 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-a[71484]: ts=2024-09-06T14:25:39.419Z caller=node_exporter.go:117 level=info collector=zfs 2024-09-06T14:25:39.714 INFO:journalctl@ceph.node-exporter.a.smithi142.stdout:Sep 06 14:25:39 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-a[71484]: ts=2024-09-06T14:25:39.419Z caller=tls_config.go:274 level=info msg="Listening on" address=[::]:9100 2024-09-06T14:25:39.714 INFO:journalctl@ceph.node-exporter.a.smithi142.stdout:Sep 06 14:25:39 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-a[71484]: ts=2024-09-06T14:25:39.419Z caller=tls_config.go:277 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-09-06T14:25:39.714 INFO:journalctl@ceph.node-exporter.a.smithi142.stdout:Sep 06 14:25:39 smithi142 bash[71392]: ffc6b152297c8bf3e77095156e3b4daf087ef3537f23553b3dd13ca97a39f587 2024-09-06T14:25:39.714 INFO:journalctl@ceph.node-exporter.a.smithi142.stdout:Sep 06 14:25:39 smithi142 systemd[1]: Started Ceph node-exporter.a for ce32462e-6c5a-11ef-bce4-c7b262605968. 2024-09-06T14:25:40.692 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph health --format=json 2024-09-06T14:25:40.797 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:40 smithi162 ceph-mon[33787]: pgmap v25: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:25:40.797 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:40 smithi162 ceph-mon[33787]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:40.797 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:40 smithi162 ceph-mon[33787]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:40.797 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:40 smithi162 ceph-mon[33787]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:40.797 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:40 smithi162 ceph-mon[33787]: Deploying daemon node-exporter.b on smithi162 2024-09-06T14:25:40.798 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 06 14:25:40 smithi162 bash[63036]: Trying to pull quay.io/prometheus/node-exporter:v1.7.0... 2024-09-06T14:25:40.953 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:25:40.958 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:40 smithi142 ceph-mon[29958]: pgmap v25: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:25:40.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:40 smithi142 ceph-mon[29958]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:40.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:40 smithi142 ceph-mon[29958]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:40.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:40 smithi142 ceph-mon[29958]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:40.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:40 smithi142 ceph-mon[29958]: Deploying daemon node-exporter.b on smithi162 2024-09-06T14:25:40.959 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:40 smithi142 ceph-mon[37372]: pgmap v25: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:25:40.959 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:40 smithi142 ceph-mon[37372]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:40.959 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:40 smithi142 ceph-mon[37372]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:40.959 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:40 smithi142 ceph-mon[37372]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:40.959 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:40 smithi142 ceph-mon[37372]: Deploying daemon node-exporter.b on smithi162 2024-09-06T14:25:41.297 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 06 14:25:40 smithi162 bash[63036]: Getting image source signatures 2024-09-06T14:25:41.297 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 06 14:25:40 smithi162 bash[63036]: Copying blob sha256:324153f2810a9927fcce320af9e4e291e0b6e805cbdd1f338386c756b9defa24 2024-09-06T14:25:41.297 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 06 14:25:40 smithi162 bash[63036]: Copying blob sha256:2abcce694348cd2c949c0e98a7400ebdfd8341021bcf6b541bc72033ce982510 2024-09-06T14:25:41.297 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 06 14:25:40 smithi162 bash[63036]: Copying blob sha256:455fd88e5221bc1e278ef2d059cd70e4df99a24e5af050ede621534276f6cf9a 2024-09-06T14:25:41.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:41.426+0000 7f62de17c640 1 -- 172.21.15.142:0/1647678670 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f62d81042f0 msgr2=0x7f62d810c8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:25:41.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:41.426+0000 7f62de17c640 1 --2- 172.21.15.142:0/1647678670 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f62d81042f0 0x7f62d810c8d0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f62c0009900 tx=0x7f62c002a410 comp rx=0 tx=0).stop 2024-09-06T14:25:41.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:41.426+0000 7f62de17c640 1 -- 172.21.15.142:0/1647678670 shutdown_connections 2024-09-06T14:25:41.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:41.426+0000 7f62de17c640 1 --2- 172.21.15.142:0/1647678670 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f62d810cea0 0x7f62d810f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:41.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:41.426+0000 7f62de17c640 1 --2- 172.21.15.142:0/1647678670 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f62d81042f0 0x7f62d810c8d0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:41.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:41.426+0000 7f62de17c640 1 --2- 172.21.15.142:0/1647678670 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f62d81038f0 0x7f62d8103cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:41.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:41.426+0000 7f62de17c640 1 -- 172.21.15.142:0/1647678670 >> 172.21.15.142:0/1647678670 conn(0x7f62d80fd440 msgr2=0x7f62d80ff860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:25:41.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:41.426+0000 7f62de17c640 1 -- 172.21.15.142:0/1647678670 shutdown_connections 2024-09-06T14:25:41.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:41.426+0000 7f62de17c640 1 -- 172.21.15.142:0/1647678670 wait complete. 2024-09-06T14:25:41.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:41.426+0000 7f62de17c640 1 Processor -- start 2024-09-06T14:25:41.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:41.427+0000 7f62de17c640 1 -- start start 2024-09-06T14:25:41.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:41.427+0000 7f62de17c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f62d81038f0 0x7f62d8101770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:25:41.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:41.427+0000 7f62de17c640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f62d81042f0 0x7f62d8101cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:25:41.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:41.427+0000 7f62de17c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f62d810cea0 0x7f62d81021f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:25:41.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:41.427+0000 7f62de17c640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f62d81121d0 con 0x7f62d810cea0 2024-09-06T14:25:41.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:41.427+0000 7f62de17c640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f62d8112050 con 0x7f62d81038f0 2024-09-06T14:25:41.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:41.427+0000 7f62de17c640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7f62d8112350 con 0x7f62d81042f0 2024-09-06T14:25:41.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:41.427+0000 7f62d7fff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f62d810cea0 0x7f62d81021f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:25:41.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:41.427+0000 7f62cffff640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f62d81042f0 0x7f62d8101cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:25:41.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:41.427+0000 7f62d7fff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f62d810cea0 0x7f62d81021f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:44736/0 (socket says 172.21.15.142:44736) 2024-09-06T14:25:41.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:41.427+0000 7f62d7fff640 1 -- 172.21.15.142:0/112566503 learned_addr learned my addr 172.21.15.142:0/112566503 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:25:41.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:41.427+0000 7f62d77fe640 1 --2- 172.21.15.142:0/112566503 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f62d81038f0 0x7f62d8101770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:25:41.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:41.427+0000 7f62d7fff640 1 -- 172.21.15.142:0/112566503 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f62d81042f0 msgr2=0x7f62d8101cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:25:41.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:41.427+0000 7f62d7fff640 1 --2- 172.21.15.142:0/112566503 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f62d81042f0 0x7f62d8101cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:41.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:41.427+0000 7f62d7fff640 1 -- 172.21.15.142:0/112566503 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f62d81038f0 msgr2=0x7f62d8101770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:25:41.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:41.427+0000 7f62d7fff640 1 --2- 172.21.15.142:0/112566503 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f62d81038f0 0x7f62d8101770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:41.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:41.427+0000 7f62d7fff640 1 -- 172.21.15.142:0/112566503 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62d81a6e00 con 0x7f62d810cea0 2024-09-06T14:25:41.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:41.427+0000 7f62d7fff640 1 --2- 172.21.15.142:0/112566503 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f62d810cea0 0x7f62d81021f0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f62d000dac0 tx=0x7f62d000df80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:25:41.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:41.427+0000 7f62d57fa640 1 -- 172.21.15.142:0/112566503 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62d0017070 con 0x7f62d810cea0 2024-09-06T14:25:41.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:41.427+0000 7f62de17c640 1 -- 172.21.15.142:0/112566503 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62d81a70f0 con 0x7f62d810cea0 2024-09-06T14:25:41.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:41.428+0000 7f62d57fa640 1 -- 172.21.15.142:0/112566503 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7f62d000be30 con 0x7f62d810cea0 2024-09-06T14:25:41.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:41.428+0000 7f62d57fa640 1 -- 172.21.15.142:0/112566503 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62d0010460 con 0x7f62d810cea0 2024-09-06T14:25:41.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:41.428+0000 7f62de17c640 1 -- 172.21.15.142:0/112566503 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62d81a7600 con 0x7f62d810cea0 2024-09-06T14:25:41.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:41.428+0000 7f62de17c640 1 -- 172.21.15.142:0/112566503 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62d806b820 con 0x7f62d810cea0 2024-09-06T14:25:41.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:41.429+0000 7f62d57fa640 1 -- 172.21.15.142:0/112566503 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 22) ==== 99844+0+0 (secure 0 0 0) 0x7f62d0004020 con 0x7f62d810cea0 2024-09-06T14:25:41.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:41.430+0000 7f62d57fa640 1 --2- 172.21.15.142:0/112566503 >> v2:172.21.15.142:6800/2944545612 conn(0x7f62c4077620 0x7f62c4079ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:25:41.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:41.430+0000 7f62d77fe640 1 --2- 172.21.15.142:0/112566503 >> v2:172.21.15.142:6800/2944545612 conn(0x7f62c4077620 0x7f62c4079ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:25:41.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:41.430+0000 7f62d57fa640 1 -- 172.21.15.142:0/112566503 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 6043+0+0 (secure 0 0 0) 0x7f62d0099540 con 0x7f62d810cea0 2024-09-06T14:25:41.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:41.430+0000 7f62d77fe640 1 --2- 172.21.15.142:0/112566503 >> v2:172.21.15.142:6800/2944545612 conn(0x7f62c4077620 0x7f62c4079ae0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f62c80095f0 tx=0x7f62c8005f90 comp rx=0 tx=0).ready entity=mgr.14601 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:25:41.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:41.434+0000 7f62d57fa640 1 -- 172.21.15.142:0/112566503 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7f62d00667d0 con 0x7f62d810cea0 2024-09-06T14:25:41.623 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:41.626+0000 7f62de17c640 1 -- 172.21.15.142:0/112566503 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f62d8102aa0 con 0x7f62d810cea0 2024-09-06T14:25:41.623 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:41 smithi142 ceph-mon[29958]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-06T14:25:41.624 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:41 smithi142 ceph-mon[37372]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-06T14:25:41.624 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:41.626+0000 7f62d57fa640 1 -- 172.21.15.142:0/112566503 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+212 (secure 0 0 0) 0x7f62d8102aa0 con 0x7f62d810cea0 2024-09-06T14:25:41.624 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:25:41.624 INFO:teuthology.orchestra.run.smithi142.stdout:{"status":"HEALTH_WARN","checks":{"MGR_MODULE_DEPENDENCY":{"severity":"HEALTH_WARN","summary":{"message":"Module 'volumes' has failed dependency: No module named 'ceph.fs'","count":1},"muted":false}},"mutes":[]} 2024-09-06T14:25:41.625 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:41.628+0000 7f62de17c640 1 -- 172.21.15.142:0/112566503 >> v2:172.21.15.142:6800/2944545612 conn(0x7f62c4077620 msgr2=0x7f62c4079ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:25:41.625 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:41.628+0000 7f62de17c640 1 --2- 172.21.15.142:0/112566503 >> v2:172.21.15.142:6800/2944545612 conn(0x7f62c4077620 0x7f62c4079ae0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f62c80095f0 tx=0x7f62c8005f90 comp rx=0 tx=0).stop 2024-09-06T14:25:41.625 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:41.628+0000 7f62de17c640 1 -- 172.21.15.142:0/112566503 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f62d810cea0 msgr2=0x7f62d81021f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:25:41.625 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:41.628+0000 7f62de17c640 1 --2- 172.21.15.142:0/112566503 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f62d810cea0 0x7f62d81021f0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f62d000dac0 tx=0x7f62d000df80 comp rx=0 tx=0).stop 2024-09-06T14:25:41.626 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:41.628+0000 7f62de17c640 1 -- 172.21.15.142:0/112566503 shutdown_connections 2024-09-06T14:25:41.626 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:41.628+0000 7f62de17c640 1 --2- 172.21.15.142:0/112566503 >> v2:172.21.15.142:6800/2944545612 conn(0x7f62c4077620 0x7f62c4079ae0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:41.626 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:41.628+0000 7f62de17c640 1 --2- 172.21.15.142:0/112566503 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f62d810cea0 0x7f62d81021f0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:41.626 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:41.628+0000 7f62de17c640 1 --2- 172.21.15.142:0/112566503 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f62d81042f0 0x7f62d8101cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:41.626 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:41.628+0000 7f62de17c640 1 --2- 172.21.15.142:0/112566503 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f62d81038f0 0x7f62d8101770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:41.626 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:41.628+0000 7f62de17c640 1 -- 172.21.15.142:0/112566503 >> 172.21.15.142:0/112566503 conn(0x7f62d80fd440 msgr2=0x7f62d81047f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:25:41.626 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:41.629+0000 7f62de17c640 1 -- 172.21.15.142:0/112566503 shutdown_connections 2024-09-06T14:25:41.626 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:41.629+0000 7f62de17c640 1 -- 172.21.15.142:0/112566503 wait complete. 2024-09-06T14:25:42.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:41 smithi162 ceph-mon[33787]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-06T14:25:42.797 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:42 smithi162 ceph-mon[33787]: pgmap v26: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:25:42.797 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:42 smithi162 ceph-mon[33787]: from='client.? 172.21.15.142:0/112566503' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:25:42.797 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 06 14:25:42 smithi162 bash[63036]: Copying config sha256:72c9c208898624938c9e4183d6686ea4a5fd3f912bc29bc3f00147924c521a3e 2024-09-06T14:25:42.798 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 06 14:25:42 smithi162 bash[63036]: Writing manifest to image destination 2024-09-06T14:25:42.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:42 smithi142 ceph-mon[29958]: pgmap v26: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:25:42.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:42 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/112566503' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:25:42.959 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:42 smithi142 ceph-mon[37372]: pgmap v26: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:25:42.959 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:42 smithi142 ceph-mon[37372]: from='client.? 172.21.15.142:0/112566503' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:25:43.297 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 06 14:25:42 smithi162 podman[63036]: 2024-09-06 14:25:42.768414502 +0000 UTC m=+2.273851916 image pull 72c9c208898624938c9e4183d6686ea4a5fd3f912bc29bc3f00147924c521a3e quay.io/prometheus/node-exporter:v1.7.0 2024-09-06T14:25:44.047 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 06 14:25:43 smithi162 podman[63036]: 2024-09-06 14:25:43.602925746 +0000 UTC m=+3.108363149 container create 07c3262277d3db43782461c7d7ede892cf9ab1a9fc01326948577fbef761c358 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-06T14:25:44.709 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:44 smithi142 ceph-mon[29958]: pgmap v27: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:25:44.709 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:44 smithi142 ceph-mon[37372]: pgmap v27: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:25:44.709 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:25:44 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: ::ffff:172.21.15.162 - - [06/Sep/2024:14:25:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-06T14:25:45.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:44 smithi162 ceph-mon[33787]: pgmap v27: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:25:45.278 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph health --format=json 2024-09-06T14:25:45.539 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:25:46.026 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:46.021+0000 7fb8cedc0640 1 -- 172.21.15.142:0/3259436954 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb8c8107d50 msgr2=0x7fb8c810e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:25:46.026 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:46.021+0000 7fb8cedc0640 1 --2- 172.21.15.142:0/3259436954 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb8c8107d50 0x7fb8c810e550 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb8c0009ca0 tx=0x7fb8c002a8c0 comp rx=0 tx=0).stop 2024-09-06T14:25:46.026 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:46.021+0000 7fb8cedc0640 1 -- 172.21.15.142:0/3259436954 shutdown_connections 2024-09-06T14:25:46.026 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:46.021+0000 7fb8cedc0640 1 --2- 172.21.15.142:0/3259436954 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb8c8107d50 0x7fb8c810e550 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:46.026 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:46.021+0000 7fb8cedc0640 1 --2- 172.21.15.142:0/3259436954 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb8c81073d0 0x7fb8c8107810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:46.026 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:46.021+0000 7fb8cedc0640 1 --2- 172.21.15.142:0/3259436954 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb8c81019c0 0x7fb8c8101da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:46.026 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:46.021+0000 7fb8cedc0640 1 -- 172.21.15.142:0/3259436954 >> 172.21.15.142:0/3259436954 conn(0x7fb8c80fd430 msgr2=0x7fb8c80ff850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:25:46.026 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:46.021+0000 7fb8cedc0640 1 -- 172.21.15.142:0/3259436954 shutdown_connections 2024-09-06T14:25:46.027 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:46.021+0000 7fb8cedc0640 1 -- 172.21.15.142:0/3259436954 wait complete. 2024-09-06T14:25:46.027 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:46.022+0000 7fb8cedc0640 1 Processor -- start 2024-09-06T14:25:46.027 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:46.022+0000 7fb8cedc0640 1 -- start start 2024-09-06T14:25:46.027 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:46.022+0000 7fb8cedc0640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb8c81019c0 0x7fb8c81986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:25:46.027 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:46.022+0000 7fb8cedc0640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb8c81073d0 0x7fb8c8198c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:25:46.027 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:46.022+0000 7fb8cedc0640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb8c8107d50 0x7fb8c819dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:25:46.027 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:46.022+0000 7fb8cedc0640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fb8c8111120 con 0x7fb8c81073d0 2024-09-06T14:25:46.027 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:46.022+0000 7fb8cedc0640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb8c8110fa0 con 0x7fb8c8107d50 2024-09-06T14:25:46.027 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:46.022+0000 7fb8cedc0640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7fb8c81112a0 con 0x7fb8c81019c0 2024-09-06T14:25:46.027 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:46.022+0000 7fb8ccb5c640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb8c81019c0 0x7fb8c81986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:25:46.027 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:46.022+0000 7fb8c7fff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb8c81073d0 0x7fb8c8198c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:25:46.027 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:46.022+0000 7fb8ccb5c640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb8c81019c0 0x7fb8c81986f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:44658/0 (socket says 172.21.15.142:44658) 2024-09-06T14:25:46.027 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:46.022+0000 7fb8ccb5c640 1 -- 172.21.15.142:0/1576905469 learned_addr learned my addr 172.21.15.142:0/1576905469 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:25:46.027 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:46.022+0000 7fb8cd35d640 1 --2- 172.21.15.142:0/1576905469 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb8c8107d50 0x7fb8c819dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:25:46.028 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:46.022+0000 7fb8ccb5c640 1 -- 172.21.15.142:0/1576905469 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb8c8107d50 msgr2=0x7fb8c819dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:25:46.028 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:46.022+0000 7fb8ccb5c640 1 --2- 172.21.15.142:0/1576905469 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb8c8107d50 0x7fb8c819dfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:46.028 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:46.022+0000 7fb8ccb5c640 1 -- 172.21.15.142:0/1576905469 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb8c81073d0 msgr2=0x7fb8c8198c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:25:46.028 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:46.022+0000 7fb8ccb5c640 1 --2- 172.21.15.142:0/1576905469 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb8c81073d0 0x7fb8c8198c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:46.028 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:46.022+0000 7fb8ccb5c640 1 -- 172.21.15.142:0/1576905469 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8c819e7a0 con 0x7fb8c81019c0 2024-09-06T14:25:46.028 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:46.022+0000 7fb8c7fff640 1 --2- 172.21.15.142:0/1576905469 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb8c81073d0 0x7fb8c8198c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-06T14:25:46.028 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:46.023+0000 7fb8ccb5c640 1 --2- 172.21.15.142:0/1576905469 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb8c81019c0 0x7fb8c81986f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb8bc00ea60 tx=0x7fb8bc00ee60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:25:46.028 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:46.023+0000 7fb8c5ffb640 1 -- 172.21.15.142:0/1576905469 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8bc00cd20 con 0x7fb8c81019c0 2024-09-06T14:25:46.028 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:46.023+0000 7fb8cedc0640 1 -- 172.21.15.142:0/1576905469 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8c819ea90 con 0x7fb8c81019c0 2024-09-06T14:25:46.028 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:46.023+0000 7fb8c5ffb640 1 -- 172.21.15.142:0/1576905469 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7fb8bc0035f0 con 0x7fb8c81019c0 2024-09-06T14:25:46.028 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:46.023+0000 7fb8c5ffb640 1 -- 172.21.15.142:0/1576905469 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8bc0044a0 con 0x7fb8c81019c0 2024-09-06T14:25:46.028 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:46.023+0000 7fb8cedc0640 1 -- 172.21.15.142:0/1576905469 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8c81a51d0 con 0x7fb8c81019c0 2024-09-06T14:25:46.028 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:46.024+0000 7fb8cedc0640 1 -- 172.21.15.142:0/1576905469 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8c8101da0 con 0x7fb8c81019c0 2024-09-06T14:25:46.028 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:46.024+0000 7fb8c5ffb640 1 -- 172.21.15.142:0/1576905469 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 22) ==== 99844+0+0 (secure 0 0 0) 0x7fb8bc010670 con 0x7fb8c81019c0 2024-09-06T14:25:46.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:46.025+0000 7fb8c5ffb640 1 --2- 172.21.15.142:0/1576905469 >> v2:172.21.15.142:6800/2944545612 conn(0x7fb8b8077620 0x7fb8b8079ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:25:46.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:46.025+0000 7fb8c5ffb640 1 -- 172.21.15.142:0/1576905469 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(62..62 src has 1..62) ==== 6043+0+0 (secure 0 0 0) 0x7fb8bc099cf0 con 0x7fb8c81019c0 2024-09-06T14:25:46.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:46.025+0000 7fb8c7fff640 1 --2- 172.21.15.142:0/1576905469 >> v2:172.21.15.142:6800/2944545612 conn(0x7fb8b8077620 0x7fb8b8079ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:25:46.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:46.025+0000 7fb8c7fff640 1 --2- 172.21.15.142:0/1576905469 >> v2:172.21.15.142:6800/2944545612 conn(0x7fb8b8077620 0x7fb8b8079ae0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fb8c8103b40 tx=0x7fb8b4009110 comp rx=0 tx=0).ready entity=mgr.14601 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:25:46.030 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:46.033+0000 7fb8c5ffb640 1 -- 172.21.15.142:0/1576905469 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7fb8bc063c20 con 0x7fb8c81019c0 2024-09-06T14:25:46.047 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 06 14:25:45 smithi162 podman[63036]: 2024-09-06 14:25:45.670085583 +0000 UTC m=+5.175522995 container init 07c3262277d3db43782461c7d7ede892cf9ab1a9fc01326948577fbef761c358 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-06T14:25:46.047 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 06 14:25:45 smithi162 podman[63036]: 2024-09-06 14:25:45.673950706 +0000 UTC m=+5.179388110 container start 07c3262277d3db43782461c7d7ede892cf9ab1a9fc01326948577fbef761c358 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-06T14:25:46.047 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 06 14:25:45 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-b[63110]: ts=2024-09-06T14:25:45.678Z caller=node_exporter.go:192 level=info msg="Starting node_exporter" version="(version=1.7.0, branch=HEAD, revision=7333465abf9efba81876303bb57e6fadb946041b)" 2024-09-06T14:25:46.048 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 06 14:25:45 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-b[63110]: ts=2024-09-06T14:25:45.678Z caller=node_exporter.go:193 level=info msg="Build context" build_context="(go=go1.21.4, platform=linux/amd64, user=root@35918982f6d8, date=20231112-23:53:35, tags=netgo osusergo static_build)" 2024-09-06T14:25:46.048 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 06 14:25:45 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-b[63110]: ts=2024-09-06T14:25:45.678Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-09-06T14:25:46.048 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 06 14:25:45 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-b[63110]: ts=2024-09-06T14:25:45.678Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-09-06T14:25:46.048 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 06 14:25:45 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-b[63110]: ts=2024-09-06T14:25:45.679Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-09-06T14:25:46.048 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 06 14:25:45 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-b[63110]: ts=2024-09-06T14:25:45.679Z caller=diskstats_linux.go:265 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-09-06T14:25:46.048 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 06 14:25:45 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-b[63110]: ts=2024-09-06T14:25:45.680Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-09-06T14:25:46.048 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 06 14:25:45 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-b[63110]: ts=2024-09-06T14:25:45.680Z caller=node_exporter.go:117 level=info collector=arp 2024-09-06T14:25:46.048 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 06 14:25:45 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-b[63110]: ts=2024-09-06T14:25:45.680Z caller=node_exporter.go:117 level=info collector=bcache 2024-09-06T14:25:46.048 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 06 14:25:45 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-b[63110]: ts=2024-09-06T14:25:45.680Z caller=node_exporter.go:117 level=info collector=bonding 2024-09-06T14:25:46.048 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 06 14:25:45 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-b[63110]: ts=2024-09-06T14:25:45.680Z caller=node_exporter.go:117 level=info collector=btrfs 2024-09-06T14:25:46.048 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 06 14:25:45 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-b[63110]: ts=2024-09-06T14:25:45.680Z caller=node_exporter.go:117 level=info collector=conntrack 2024-09-06T14:25:46.048 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 06 14:25:45 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-b[63110]: ts=2024-09-06T14:25:45.680Z caller=node_exporter.go:117 level=info collector=cpu 2024-09-06T14:25:46.048 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 06 14:25:45 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-b[63110]: ts=2024-09-06T14:25:45.680Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-09-06T14:25:46.049 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 06 14:25:45 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-b[63110]: ts=2024-09-06T14:25:45.680Z caller=node_exporter.go:117 level=info collector=diskstats 2024-09-06T14:25:46.049 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 06 14:25:45 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-b[63110]: ts=2024-09-06T14:25:45.680Z caller=node_exporter.go:117 level=info collector=dmi 2024-09-06T14:25:46.049 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 06 14:25:45 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-b[63110]: ts=2024-09-06T14:25:45.680Z caller=node_exporter.go:117 level=info collector=edac 2024-09-06T14:25:46.049 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 06 14:25:45 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-b[63110]: ts=2024-09-06T14:25:45.680Z caller=node_exporter.go:117 level=info collector=entropy 2024-09-06T14:25:46.049 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 06 14:25:45 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-b[63110]: ts=2024-09-06T14:25:45.680Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-09-06T14:25:46.049 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 06 14:25:45 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-b[63110]: ts=2024-09-06T14:25:45.680Z caller=node_exporter.go:117 level=info collector=filefd 2024-09-06T14:25:46.050 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 06 14:25:45 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-b[63110]: ts=2024-09-06T14:25:45.680Z caller=node_exporter.go:117 level=info collector=filesystem 2024-09-06T14:25:46.050 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 06 14:25:45 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-b[63110]: ts=2024-09-06T14:25:45.680Z caller=node_exporter.go:117 level=info collector=hwmon 2024-09-06T14:25:46.050 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 06 14:25:45 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-b[63110]: ts=2024-09-06T14:25:45.680Z caller=node_exporter.go:117 level=info collector=infiniband 2024-09-06T14:25:46.050 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 06 14:25:45 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-b[63110]: ts=2024-09-06T14:25:45.680Z caller=node_exporter.go:117 level=info collector=ipvs 2024-09-06T14:25:46.050 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 06 14:25:45 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-b[63110]: ts=2024-09-06T14:25:45.680Z caller=node_exporter.go:117 level=info collector=loadavg 2024-09-06T14:25:46.050 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 06 14:25:45 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-b[63110]: ts=2024-09-06T14:25:45.680Z caller=node_exporter.go:117 level=info collector=mdadm 2024-09-06T14:25:46.050 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 06 14:25:45 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-b[63110]: ts=2024-09-06T14:25:45.680Z caller=node_exporter.go:117 level=info collector=meminfo 2024-09-06T14:25:46.050 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 06 14:25:45 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-b[63110]: ts=2024-09-06T14:25:45.680Z caller=node_exporter.go:117 level=info collector=netclass 2024-09-06T14:25:46.050 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 06 14:25:45 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-b[63110]: ts=2024-09-06T14:25:45.680Z caller=node_exporter.go:117 level=info collector=netdev 2024-09-06T14:25:46.050 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 06 14:25:45 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-b[63110]: ts=2024-09-06T14:25:45.680Z caller=node_exporter.go:117 level=info collector=netstat 2024-09-06T14:25:46.050 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 06 14:25:45 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-b[63110]: ts=2024-09-06T14:25:45.680Z caller=node_exporter.go:117 level=info collector=nfs 2024-09-06T14:25:46.050 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 06 14:25:45 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-b[63110]: ts=2024-09-06T14:25:45.680Z caller=node_exporter.go:117 level=info collector=nfsd 2024-09-06T14:25:46.050 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 06 14:25:45 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-b[63110]: ts=2024-09-06T14:25:45.680Z caller=node_exporter.go:117 level=info collector=nvme 2024-09-06T14:25:46.050 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 06 14:25:45 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-b[63110]: ts=2024-09-06T14:25:45.680Z caller=node_exporter.go:117 level=info collector=os 2024-09-06T14:25:46.050 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 06 14:25:45 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-b[63110]: ts=2024-09-06T14:25:45.680Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-09-06T14:25:46.051 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 06 14:25:45 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-b[63110]: ts=2024-09-06T14:25:45.680Z caller=node_exporter.go:117 level=info collector=pressure 2024-09-06T14:25:46.051 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 06 14:25:45 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-b[63110]: ts=2024-09-06T14:25:45.680Z caller=node_exporter.go:117 level=info collector=rapl 2024-09-06T14:25:46.051 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 06 14:25:45 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-b[63110]: ts=2024-09-06T14:25:45.680Z caller=node_exporter.go:117 level=info collector=schedstat 2024-09-06T14:25:46.051 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 06 14:25:45 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-b[63110]: ts=2024-09-06T14:25:45.680Z caller=node_exporter.go:117 level=info collector=selinux 2024-09-06T14:25:46.051 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 06 14:25:45 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-b[63110]: ts=2024-09-06T14:25:45.680Z caller=node_exporter.go:117 level=info collector=sockstat 2024-09-06T14:25:46.051 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 06 14:25:45 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-b[63110]: ts=2024-09-06T14:25:45.680Z caller=node_exporter.go:117 level=info collector=softnet 2024-09-06T14:25:46.051 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 06 14:25:45 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-b[63110]: ts=2024-09-06T14:25:45.680Z caller=node_exporter.go:117 level=info collector=stat 2024-09-06T14:25:46.051 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 06 14:25:45 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-b[63110]: ts=2024-09-06T14:25:45.680Z caller=node_exporter.go:117 level=info collector=tapestats 2024-09-06T14:25:46.051 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 06 14:25:45 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-b[63110]: ts=2024-09-06T14:25:45.680Z caller=node_exporter.go:117 level=info collector=textfile 2024-09-06T14:25:46.051 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 06 14:25:45 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-b[63110]: ts=2024-09-06T14:25:45.680Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-09-06T14:25:46.051 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 06 14:25:45 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-b[63110]: ts=2024-09-06T14:25:45.680Z caller=node_exporter.go:117 level=info collector=time 2024-09-06T14:25:46.051 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 06 14:25:45 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-b[63110]: ts=2024-09-06T14:25:45.680Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-09-06T14:25:46.051 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 06 14:25:45 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-b[63110]: ts=2024-09-06T14:25:45.680Z caller=node_exporter.go:117 level=info collector=uname 2024-09-06T14:25:46.051 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 06 14:25:45 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-b[63110]: ts=2024-09-06T14:25:45.680Z caller=node_exporter.go:117 level=info collector=vmstat 2024-09-06T14:25:46.051 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 06 14:25:45 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-b[63110]: ts=2024-09-06T14:25:45.680Z caller=node_exporter.go:117 level=info collector=xfs 2024-09-06T14:25:46.052 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 06 14:25:45 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-b[63110]: ts=2024-09-06T14:25:45.680Z caller=node_exporter.go:117 level=info collector=zfs 2024-09-06T14:25:46.052 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 06 14:25:45 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-b[63110]: ts=2024-09-06T14:25:45.681Z caller=tls_config.go:274 level=info msg="Listening on" address=[::]:9100 2024-09-06T14:25:46.052 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 06 14:25:45 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-b[63110]: ts=2024-09-06T14:25:45.681Z caller=tls_config.go:277 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-09-06T14:25:46.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:46.230+0000 7fb8cedc0640 1 -- 172.21.15.142:0/1576905469 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fb8c81a5490 con 0x7fb8c81019c0 2024-09-06T14:25:46.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:46.231+0000 7fb8c5ffb640 1 -- 172.21.15.142:0/1576905469 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+212 (secure 0 0 0) 0x7fb8bc06c0d0 con 0x7fb8c81019c0 2024-09-06T14:25:46.229 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:25:46.229 INFO:teuthology.orchestra.run.smithi142.stdout:{"status":"HEALTH_WARN","checks":{"MGR_MODULE_DEPENDENCY":{"severity":"HEALTH_WARN","summary":{"message":"Module 'volumes' has failed dependency: No module named 'ceph.fs'","count":1},"muted":false}},"mutes":[]} 2024-09-06T14:25:46.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:46.233+0000 7fb8cedc0640 1 -- 172.21.15.142:0/1576905469 >> v2:172.21.15.142:6800/2944545612 conn(0x7fb8b8077620 msgr2=0x7fb8b8079ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:25:46.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:46.233+0000 7fb8cedc0640 1 --2- 172.21.15.142:0/1576905469 >> v2:172.21.15.142:6800/2944545612 conn(0x7fb8b8077620 0x7fb8b8079ae0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fb8c8103b40 tx=0x7fb8b4009110 comp rx=0 tx=0).stop 2024-09-06T14:25:46.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:46.233+0000 7fb8cedc0640 1 -- 172.21.15.142:0/1576905469 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb8c81019c0 msgr2=0x7fb8c81986f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:25:46.232 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:46.233+0000 7fb8cedc0640 1 --2- 172.21.15.142:0/1576905469 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb8c81019c0 0x7fb8c81986f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb8bc00ea60 tx=0x7fb8bc00ee60 comp rx=0 tx=0).stop 2024-09-06T14:25:46.232 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:46.233+0000 7fb8cedc0640 1 -- 172.21.15.142:0/1576905469 shutdown_connections 2024-09-06T14:25:46.232 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:46.233+0000 7fb8cedc0640 1 --2- 172.21.15.142:0/1576905469 >> v2:172.21.15.142:6800/2944545612 conn(0x7fb8b8077620 0x7fb8b8079ae0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:46.232 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:46.233+0000 7fb8cedc0640 1 --2- 172.21.15.142:0/1576905469 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb8c8107d50 0x7fb8c819dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:46.232 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:46.233+0000 7fb8cedc0640 1 --2- 172.21.15.142:0/1576905469 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb8c81073d0 0x7fb8c8198c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:46.232 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:46.233+0000 7fb8cedc0640 1 --2- 172.21.15.142:0/1576905469 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb8c81019c0 0x7fb8c81986f0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:46.232 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:46.233+0000 7fb8cedc0640 1 -- 172.21.15.142:0/1576905469 >> 172.21.15.142:0/1576905469 conn(0x7fb8c80fd430 msgr2=0x7fb8c81044d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:25:46.232 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:46.233+0000 7fb8cedc0640 1 -- 172.21.15.142:0/1576905469 shutdown_connections 2024-09-06T14:25:46.232 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:46.233+0000 7fb8cedc0640 1 -- 172.21.15.142:0/1576905469 wait complete. 2024-09-06T14:25:46.797 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:46 smithi162 ceph-mon[33787]: pgmap v28: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:25:46.797 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:46 smithi162 ceph-mon[33787]: from='client.? 172.21.15.142:0/1576905469' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:25:46.797 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 06 14:25:46 smithi162 bash[63036]: 07c3262277d3db43782461c7d7ede892cf9ab1a9fc01326948577fbef761c358 2024-09-06T14:25:46.797 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 06 14:25:46 smithi162 systemd[1]: Started Ceph node-exporter.b for ce32462e-6c5a-11ef-bce4-c7b262605968. 2024-09-06T14:25:46.959 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:46 smithi142 ceph-mon[37372]: pgmap v28: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:25:46.959 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:46 smithi142 ceph-mon[37372]: from='client.? 172.21.15.142:0/1576905469' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:25:46.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:46 smithi142 ceph-mon[29958]: pgmap v28: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:25:46.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:46 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/1576905469' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:25:47.315 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 06 14:25:47 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-iscsi-iscsi-a[59853]: debug there is no tcmu-runner data available 2024-09-06T14:25:47.729 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:47 smithi142 ceph-mon[37372]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:47.729 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:47 smithi142 ceph-mon[37372]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:47.729 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:47 smithi142 ceph-mon[37372]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:47.729 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:47 smithi142 ceph-mon[37372]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:47.729 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:47 smithi142 ceph-mon[37372]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:47.729 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:47 smithi142 ceph-mon[37372]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-06T14:25:47.730 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:47 smithi142 ceph-mon[37372]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-06T14:25:47.730 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:47 smithi142 ceph-mon[29958]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:47.730 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:47 smithi142 ceph-mon[29958]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:47.730 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:47 smithi142 ceph-mon[29958]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:47.730 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:47 smithi142 ceph-mon[29958]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:47.730 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:47 smithi142 ceph-mon[29958]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:47.730 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:47 smithi142 ceph-mon[29958]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-06T14:25:47.730 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:47 smithi142 ceph-mon[29958]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-06T14:25:48.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:47 smithi162 ceph-mon[33787]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:48.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:47 smithi162 ceph-mon[33787]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:48.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:47 smithi162 ceph-mon[33787]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:48.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:47 smithi162 ceph-mon[33787]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:48.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:47 smithi162 ceph-mon[33787]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:48.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:47 smithi162 ceph-mon[33787]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-06T14:25:48.048 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:47 smithi162 ceph-mon[33787]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-06T14:25:48.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:48 smithi142 ceph-mon[29958]: pgmap v29: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:25:48.959 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:48 smithi142 ceph-mon[37372]: pgmap v29: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:25:49.032 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:48 smithi162 ceph-mon[33787]: pgmap v29: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:25:50.036 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph health --format=json 2024-09-06T14:25:50.284 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:25:50.741 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:50.743+0000 7f1dfab2a640 1 -- 172.21.15.142:0/1238613215 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1df4107d50 msgr2=0x7f1df410e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:25:50.741 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:50.743+0000 7f1dfab2a640 1 --2- 172.21.15.142:0/1238613215 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1df4107d50 0x7f1df410e550 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f1dec009be0 tx=0x7f1dec02a6c0 comp rx=0 tx=0).stop 2024-09-06T14:25:50.741 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:50.743+0000 7f1dfab2a640 1 -- 172.21.15.142:0/1238613215 shutdown_connections 2024-09-06T14:25:50.741 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:50.743+0000 7f1dfab2a640 1 --2- 172.21.15.142:0/1238613215 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1df4107d50 0x7f1df410e550 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:50.741 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:50.743+0000 7f1dfab2a640 1 --2- 172.21.15.142:0/1238613215 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1df41073d0 0x7f1df4107810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:50.741 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:50.743+0000 7f1dfab2a640 1 --2- 172.21.15.142:0/1238613215 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1df41019c0 0x7f1df4101da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:50.741 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:50.743+0000 7f1dfab2a640 1 -- 172.21.15.142:0/1238613215 >> 172.21.15.142:0/1238613215 conn(0x7f1df40fd430 msgr2=0x7f1df40ff850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:25:50.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:50.743+0000 7f1dfab2a640 1 -- 172.21.15.142:0/1238613215 shutdown_connections 2024-09-06T14:25:50.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:50.743+0000 7f1dfab2a640 1 -- 172.21.15.142:0/1238613215 wait complete. 2024-09-06T14:25:50.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:50.744+0000 7f1dfab2a640 1 Processor -- start 2024-09-06T14:25:50.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:50.744+0000 7f1dfab2a640 1 -- start start 2024-09-06T14:25:50.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:50.744+0000 7f1dfab2a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1df41019c0 0x7f1df4197290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:25:50.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:50.744+0000 7f1dfab2a640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1df41073d0 0x7f1df41977d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:25:50.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:50.744+0000 7f1dfab2a640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1df4107d50 0x7f1df4197d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:25:50.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:50.744+0000 7f1dfab2a640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f1df41110e0 con 0x7f1df41073d0 2024-09-06T14:25:50.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:50.744+0000 7f1dfab2a640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1df4110f60 con 0x7f1df41019c0 2024-09-06T14:25:50.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:50.744+0000 7f1dfab2a640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7f1df4111260 con 0x7f1df4107d50 2024-09-06T14:25:50.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:50.745+0000 7f1df3fff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1df41073d0 0x7f1df41977d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:25:50.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:50.745+0000 7f1df90c7640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1df4107d50 0x7f1df4197d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:25:50.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:50.745+0000 7f1df3fff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1df41073d0 0x7f1df41977d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:44302/0 (socket says 172.21.15.142:44302) 2024-09-06T14:25:50.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:50.745+0000 7f1df3fff640 1 -- 172.21.15.142:0/2326674886 learned_addr learned my addr 172.21.15.142:0/2326674886 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:25:50.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:50.745+0000 7f1df88c6640 1 --2- 172.21.15.142:0/2326674886 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1df41019c0 0x7f1df4197290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:25:50.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:50.745+0000 7f1df3fff640 1 -- 172.21.15.142:0/2326674886 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1df4107d50 msgr2=0x7f1df4197d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:25:50.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:50.745+0000 7f1df3fff640 1 --2- 172.21.15.142:0/2326674886 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1df4107d50 0x7f1df4197d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:50.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:50.745+0000 7f1df3fff640 1 -- 172.21.15.142:0/2326674886 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1df41019c0 msgr2=0x7f1df4197290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:25:50.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:50.745+0000 7f1df3fff640 1 --2- 172.21.15.142:0/2326674886 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1df41019c0 0x7f1df4197290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:50.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:50.745+0000 7f1df3fff640 1 -- 172.21.15.142:0/2326674886 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1df41a6a70 con 0x7f1df41073d0 2024-09-06T14:25:50.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:50.745+0000 7f1df90c7640 1 --2- 172.21.15.142:0/2326674886 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1df4107d50 0x7f1df4197d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-06T14:25:50.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:50.745+0000 7f1df3fff640 1 --2- 172.21.15.142:0/2326674886 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1df41073d0 0x7f1df41977d0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f1de000d950 tx=0x7f1de000dd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:25:50.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:50.745+0000 7f1df1ffb640 1 -- 172.21.15.142:0/2326674886 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1de0017070 con 0x7f1df41073d0 2024-09-06T14:25:50.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:50.745+0000 7f1df1ffb640 1 -- 172.21.15.142:0/2326674886 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7f1de000de80 con 0x7f1df41073d0 2024-09-06T14:25:50.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:50.745+0000 7f1dfab2a640 1 -- 172.21.15.142:0/2326674886 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1df41a6d60 con 0x7f1df41073d0 2024-09-06T14:25:50.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:50.745+0000 7f1df1ffb640 1 -- 172.21.15.142:0/2326674886 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1de0010e60 con 0x7f1df41073d0 2024-09-06T14:25:50.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:50.746+0000 7f1dfab2a640 1 -- 172.21.15.142:0/2326674886 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1df41a7260 con 0x7f1df41073d0 2024-09-06T14:25:50.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:50.747+0000 7f1dfab2a640 1 -- 172.21.15.142:0/2326674886 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1df4101da0 con 0x7f1df41073d0 2024-09-06T14:25:50.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:50.747+0000 7f1df1ffb640 1 -- 172.21.15.142:0/2326674886 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 22) ==== 99844+0+0 (secure 0 0 0) 0x7f1de0010460 con 0x7f1df41073d0 2024-09-06T14:25:50.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:50.748+0000 7f1df1ffb640 1 --2- 172.21.15.142:0/2326674886 >> v2:172.21.15.142:6800/2944545612 conn(0x7f1de4077620 0x7f1de4079ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:25:50.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:50.748+0000 7f1df1ffb640 1 -- 172.21.15.142:0/2326674886 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 6043+0+0 (secure 0 0 0) 0x7f1de00994c0 con 0x7f1df41073d0 2024-09-06T14:25:50.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:50.748+0000 7f1df88c6640 1 --2- 172.21.15.142:0/2326674886 >> v2:172.21.15.142:6800/2944545612 conn(0x7f1de4077620 0x7f1de4079ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:25:50.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:50.748+0000 7f1df88c6640 1 --2- 172.21.15.142:0/2326674886 >> v2:172.21.15.142:6800/2944545612 conn(0x7f1de4077620 0x7f1de4079ae0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f1de8005fd0 tx=0x7f1de8009260 comp rx=0 tx=0).ready entity=mgr.14601 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:25:50.749 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:50.752+0000 7f1df1ffb640 1 -- 172.21.15.142:0/2326674886 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7f1de0066730 con 0x7f1df41073d0 2024-09-06T14:25:50.946 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:50.949+0000 7f1dfab2a640 1 -- 172.21.15.142:0/2326674886 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f1df4109b30 con 0x7f1df41073d0 2024-09-06T14:25:50.947 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:50 smithi142 ceph-mon[37372]: pgmap v30: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:25:50.947 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:50 smithi142 ceph-mon[37372]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:50.947 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:50 smithi142 ceph-mon[37372]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:50.947 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:50 smithi142 ceph-mon[37372]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:50.947 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:50 smithi142 ceph-mon[37372]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:50.947 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:50 smithi142 ceph-mon[37372]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:25:50.947 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:50 smithi142 ceph-mon[37372]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-06T14:25:50.947 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:50 smithi142 ceph-mon[37372]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:50.947 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:50 smithi142 ceph-mon[29958]: pgmap v30: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:25:50.948 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:50 smithi142 ceph-mon[29958]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:50.948 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:50 smithi142 ceph-mon[29958]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:50.948 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:50 smithi142 ceph-mon[29958]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:50.948 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:50 smithi142 ceph-mon[29958]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:50.948 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:50 smithi142 ceph-mon[29958]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:25:50.948 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:50 smithi142 ceph-mon[29958]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-06T14:25:50.948 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:50 smithi142 ceph-mon[29958]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:50.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:50.949+0000 7f1df1ffb640 1 -- 172.21.15.142:0/2326674886 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+212 (secure 0 0 0) 0x7f1de006b150 con 0x7f1df41073d0 2024-09-06T14:25:50.949 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:25:50.949 INFO:teuthology.orchestra.run.smithi142.stdout:{"status":"HEALTH_WARN","checks":{"MGR_MODULE_DEPENDENCY":{"severity":"HEALTH_WARN","summary":{"message":"Module 'volumes' has failed dependency: No module named 'ceph.fs'","count":1},"muted":false}},"mutes":[]} 2024-09-06T14:25:50.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:50.951+0000 7f1dfab2a640 1 -- 172.21.15.142:0/2326674886 >> v2:172.21.15.142:6800/2944545612 conn(0x7f1de4077620 msgr2=0x7f1de4079ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:25:50.950 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:50.951+0000 7f1dfab2a640 1 --2- 172.21.15.142:0/2326674886 >> v2:172.21.15.142:6800/2944545612 conn(0x7f1de4077620 0x7f1de4079ae0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f1de8005fd0 tx=0x7f1de8009260 comp rx=0 tx=0).stop 2024-09-06T14:25:50.950 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:50.951+0000 7f1dfab2a640 1 -- 172.21.15.142:0/2326674886 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1df41073d0 msgr2=0x7f1df41977d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:25:50.950 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:50.951+0000 7f1dfab2a640 1 --2- 172.21.15.142:0/2326674886 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1df41073d0 0x7f1df41977d0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f1de000d950 tx=0x7f1de000dd50 comp rx=0 tx=0).stop 2024-09-06T14:25:50.950 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:50.951+0000 7f1dfab2a640 1 -- 172.21.15.142:0/2326674886 shutdown_connections 2024-09-06T14:25:50.950 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:50.951+0000 7f1dfab2a640 1 --2- 172.21.15.142:0/2326674886 >> v2:172.21.15.142:6800/2944545612 conn(0x7f1de4077620 0x7f1de4079ae0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:50.950 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:50.952+0000 7f1dfab2a640 1 --2- 172.21.15.142:0/2326674886 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1df4107d50 0x7f1df4197d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:50.950 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:50.952+0000 7f1dfab2a640 1 --2- 172.21.15.142:0/2326674886 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1df41073d0 0x7f1df41977d0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:50.950 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:50.952+0000 7f1dfab2a640 1 --2- 172.21.15.142:0/2326674886 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1df41019c0 0x7f1df4197290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:50.950 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:50.952+0000 7f1dfab2a640 1 -- 172.21.15.142:0/2326674886 >> 172.21.15.142:0/2326674886 conn(0x7f1df40fd430 msgr2=0x7f1df40fedd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:25:50.950 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:50.952+0000 7f1dfab2a640 1 -- 172.21.15.142:0/2326674886 shutdown_connections 2024-09-06T14:25:50.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:50.952+0000 7f1dfab2a640 1 -- 172.21.15.142:0/2326674886 wait complete. 2024-09-06T14:25:51.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:50 smithi162 ceph-mon[33787]: pgmap v30: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:25:51.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:50 smithi162 ceph-mon[33787]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:51.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:50 smithi162 ceph-mon[33787]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:51.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:50 smithi162 ceph-mon[33787]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:51.298 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:50 smithi162 ceph-mon[33787]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:51.298 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:50 smithi162 ceph-mon[33787]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:25:51.298 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:50 smithi162 ceph-mon[33787]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-06T14:25:51.298 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:50 smithi162 ceph-mon[33787]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:52.046 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:51 smithi142 ceph-mon[29958]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-06T14:25:52.046 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:51 smithi142 ceph-mon[29958]: Reconfiguring daemon alertmanager.a on smithi142 2024-09-06T14:25:52.046 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:51 smithi142 ceph-mon[29958]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:52.047 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:51 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/2326674886' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:25:52.047 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:51 smithi142 ceph-mon[37372]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-06T14:25:52.047 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:51 smithi142 ceph-mon[37372]: Reconfiguring daemon alertmanager.a on smithi142 2024-09-06T14:25:52.047 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:51 smithi142 ceph-mon[37372]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:52.047 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:51 smithi142 ceph-mon[37372]: from='client.? 172.21.15.142:0/2326674886' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:25:52.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:51 smithi162 ceph-mon[33787]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-06T14:25:52.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:51 smithi162 ceph-mon[33787]: Reconfiguring daemon alertmanager.a on smithi142 2024-09-06T14:25:52.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:51 smithi162 ceph-mon[33787]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:52.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:51 smithi162 ceph-mon[33787]: from='client.? 172.21.15.142:0/2326674886' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:25:53.170 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:52 smithi142 ceph-mon[29958]: pgmap v31: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:25:53.170 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:52 smithi142 ceph-mon[37372]: pgmap v31: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:25:53.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:52 smithi162 ceph-mon[33787]: pgmap v31: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:25:54.653 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:25:54 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: ::ffff:172.21.15.162 - - [06/Sep/2024:14:25:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-06T14:25:54.828 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph health --format=json 2024-09-06T14:25:54.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:54 smithi142 ceph-mon[29958]: pgmap v32: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:25:54.959 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:54 smithi142 ceph-mon[37372]: pgmap v32: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:25:54.960 INFO:journalctl@ceph.alertmanager.a.smithi142.stdout:Sep 06 14:25:54 smithi142 systemd[1]: Stopping Ceph alertmanager.a for ce32462e-6c5a-11ef-bce4-c7b262605968... 2024-09-06T14:25:55.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:54 smithi162 ceph-mon[33787]: pgmap v32: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:25:55.416 INFO:journalctl@ceph.alertmanager.a.smithi142.stdout:Sep 06 14:25:55 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-alertmanager-a[70815]: ts=2024-09-06T14:25:55.085Z caller=main.go:542 level=info msg="Received SIGTERM, exiting gracefully..." 2024-09-06T14:25:55.416 INFO:journalctl@ceph.alertmanager.a.smithi142.stdout:Sep 06 14:25:55 smithi142 podman[72552]: 2024-09-06 14:25:55.126568947 +0000 UTC m=+0.145213223 container died 9bea55ee93597a4f79718b69728fa4198413da559e7357778dd9aa67cae822e3 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-06T14:25:55.709 INFO:journalctl@ceph.alertmanager.a.smithi142.stdout:Sep 06 14:25:55 smithi142 podman[72552]: 2024-09-06 14:25:55.535976478 +0000 UTC m=+0.554620754 container cleanup 9bea55ee93597a4f79718b69728fa4198413da559e7357778dd9aa67cae822e3 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-06T14:25:55.709 INFO:journalctl@ceph.alertmanager.a.smithi142.stdout:Sep 06 14:25:55 smithi142 bash[72552]: ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-alertmanager-a 2024-09-06T14:25:55.855 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:25:56.124 INFO:journalctl@ceph.alertmanager.a.smithi142.stdout:Sep 06 14:25:55 smithi142 podman[72589]: 2024-09-06 14:25:55.762046449 +0000 UTC m=+0.637645273 container remove 9bea55ee93597a4f79718b69728fa4198413da559e7357778dd9aa67cae822e3 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-06T14:25:56.124 INFO:journalctl@ceph.alertmanager.a.smithi142.stdout:Sep 06 14:25:55 smithi142 podman[72589]: 2024-09-06 14:25:55.8543163 +0000 UTC m=+0.729915125 volume remove 49ed157c762f9b402cee411a43398b4273cd07bf071eaebd683632a5a3d253fb 2024-09-06T14:25:56.124 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:55 smithi142 ceph-mon[29958]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]} : dispatch 2024-09-06T14:25:56.125 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:55 smithi142 ceph-mon[29958]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]} : dispatch 2024-09-06T14:25:56.125 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:55 smithi142 ceph-mon[29958]: from='mgr.14601 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]} : dispatch 2024-09-06T14:25:56.125 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:55 smithi142 ceph-mon[29958]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]} : dispatch 2024-09-06T14:25:56.125 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:55 smithi142 ceph-mon[29958]: from='mgr.14601 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]} : dispatch 2024-09-06T14:25:56.125 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:55 smithi142 ceph-mon[29958]: from='mgr.14601 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]} : dispatch 2024-09-06T14:25:56.125 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:55 smithi142 ceph-mon[37372]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]} : dispatch 2024-09-06T14:25:56.125 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:55 smithi142 ceph-mon[37372]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]} : dispatch 2024-09-06T14:25:56.125 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:55 smithi142 ceph-mon[37372]: from='mgr.14601 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]} : dispatch 2024-09-06T14:25:56.125 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:55 smithi142 ceph-mon[37372]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]} : dispatch 2024-09-06T14:25:56.126 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:55 smithi142 ceph-mon[37372]: from='mgr.14601 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]} : dispatch 2024-09-06T14:25:56.126 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:55 smithi142 ceph-mon[37372]: from='mgr.14601 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]} : dispatch 2024-09-06T14:25:56.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:55 smithi162 ceph-mon[33787]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]} : dispatch 2024-09-06T14:25:56.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:55 smithi162 ceph-mon[33787]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]} : dispatch 2024-09-06T14:25:56.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:55 smithi162 ceph-mon[33787]: from='mgr.14601 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]} : dispatch 2024-09-06T14:25:56.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:55 smithi162 ceph-mon[33787]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]} : dispatch 2024-09-06T14:25:56.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:55 smithi162 ceph-mon[33787]: from='mgr.14601 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]} : dispatch 2024-09-06T14:25:56.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:55 smithi162 ceph-mon[33787]: from='mgr.14601 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]} : dispatch 2024-09-06T14:25:56.379 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:56.353+0000 7fb3e5e19640 1 -- 172.21.15.142:0/3277151010 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb3e01019c0 msgr2=0x7fb3e0101da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:25:56.380 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:56.353+0000 7fb3e5e19640 1 --2- 172.21.15.142:0/3277151010 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb3e01019c0 0x7fb3e0101da0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fb3d0009ca0 tx=0x7fb3d002a8c0 comp rx=0 tx=0).stop 2024-09-06T14:25:56.380 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:56.353+0000 7fb3e5e19640 1 -- 172.21.15.142:0/3277151010 shutdown_connections 2024-09-06T14:25:56.380 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:56.353+0000 7fb3e5e19640 1 --2- 172.21.15.142:0/3277151010 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb3e0107d50 0x7fb3e010e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:56.380 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:56.353+0000 7fb3e5e19640 1 --2- 172.21.15.142:0/3277151010 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb3e01073d0 0x7fb3e0107810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:56.380 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:56.353+0000 7fb3e5e19640 1 --2- 172.21.15.142:0/3277151010 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb3e01019c0 0x7fb3e0101da0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:56.380 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:56.353+0000 7fb3e5e19640 1 -- 172.21.15.142:0/3277151010 >> 172.21.15.142:0/3277151010 conn(0x7fb3e00fd430 msgr2=0x7fb3e00ff850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:25:56.380 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:56.353+0000 7fb3e5e19640 1 -- 172.21.15.142:0/3277151010 shutdown_connections 2024-09-06T14:25:56.380 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:56.353+0000 7fb3e5e19640 1 -- 172.21.15.142:0/3277151010 wait complete. 2024-09-06T14:25:56.380 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:56.353+0000 7fb3e5e19640 1 Processor -- start 2024-09-06T14:25:56.380 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:56.354+0000 7fb3e5e19640 1 -- start start 2024-09-06T14:25:56.380 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:56.354+0000 7fb3e5e19640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb3e01019c0 0x7fb3e010f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:25:56.381 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:56.354+0000 7fb3e5e19640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb3e01073d0 0x7fb3e010f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:25:56.381 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:56.354+0000 7fb3e5e19640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb3e0107d50 0x7fb3e010fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:25:56.381 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:56.354+0000 7fb3e5e19640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fb3e0112220 con 0x7fb3e01019c0 2024-09-06T14:25:56.381 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:56.354+0000 7fb3e5e19640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb3e01120a0 con 0x7fb3e0107d50 2024-09-06T14:25:56.381 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:56.354+0000 7fb3e5e19640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7fb3e01123a0 con 0x7fb3e01073d0 2024-09-06T14:25:56.381 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:56.354+0000 7fb3df7fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb3e01019c0 0x7fb3e010f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:25:56.381 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:56.354+0000 7fb3deffd640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb3e01073d0 0x7fb3e010f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:25:56.381 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:56.354+0000 7fb3df7fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb3e01019c0 0x7fb3e010f060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:44314/0 (socket says 172.21.15.142:44314) 2024-09-06T14:25:56.381 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:56.354+0000 7fb3df7fe640 1 -- 172.21.15.142:0/3753771315 learned_addr learned my addr 172.21.15.142:0/3753771315 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:25:56.381 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:56.354+0000 7fb3df7fe640 1 -- 172.21.15.142:0/3753771315 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb3e01073d0 msgr2=0x7fb3e010f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:25:56.381 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:56.354+0000 7fb3df7fe640 1 --2- 172.21.15.142:0/3753771315 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb3e01073d0 0x7fb3e010f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:56.381 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:56.354+0000 7fb3df7fe640 1 -- 172.21.15.142:0/3753771315 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb3e0107d50 msgr2=0x7fb3e010fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:25:56.381 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:56.354+0000 7fb3dffff640 1 --2- 172.21.15.142:0/3753771315 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb3e0107d50 0x7fb3e010fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:25:56.382 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:56.354+0000 7fb3df7fe640 1 --2- 172.21.15.142:0/3753771315 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb3e0107d50 0x7fb3e010fae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:56.382 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:56.354+0000 7fb3df7fe640 1 -- 172.21.15.142:0/3753771315 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3e01a6b40 con 0x7fb3e01019c0 2024-09-06T14:25:56.382 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:56.354+0000 7fb3df7fe640 1 --2- 172.21.15.142:0/3753771315 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb3e01019c0 0x7fb3e010f060 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fb3d0006010 tx=0x7fb3d00093d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:25:56.382 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:56.355+0000 7fb3dcff9640 1 -- 172.21.15.142:0/3753771315 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3d0003db0 con 0x7fb3e01019c0 2024-09-06T14:25:56.382 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:56.355+0000 7fb3e5e19640 1 -- 172.21.15.142:0/3753771315 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3e01a6dd0 con 0x7fb3e01019c0 2024-09-06T14:25:56.382 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:56.355+0000 7fb3dcff9640 1 -- 172.21.15.142:0/3753771315 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7fb3d00044c0 con 0x7fb3e01019c0 2024-09-06T14:25:56.382 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:56.355+0000 7fb3dcff9640 1 -- 172.21.15.142:0/3753771315 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3d0030c20 con 0x7fb3e01019c0 2024-09-06T14:25:56.382 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:56.355+0000 7fb3e5e19640 1 -- 172.21.15.142:0/3753771315 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3e01a7300 con 0x7fb3e01019c0 2024-09-06T14:25:56.382 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:56.356+0000 7fb3e5e19640 1 -- 172.21.15.142:0/3753771315 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3e0101da0 con 0x7fb3e01019c0 2024-09-06T14:25:56.382 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:56.356+0000 7fb3dcff9640 1 -- 172.21.15.142:0/3753771315 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 22) ==== 99844+0+0 (secure 0 0 0) 0x7fb3d0004040 con 0x7fb3e01019c0 2024-09-06T14:25:56.382 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:56.357+0000 7fb3dcff9640 1 --2- 172.21.15.142:0/3753771315 >> v2:172.21.15.142:6800/2944545612 conn(0x7fb3cc077620 0x7fb3cc079ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:25:56.382 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:56.357+0000 7fb3deffd640 1 --2- 172.21.15.142:0/3753771315 >> v2:172.21.15.142:6800/2944545612 conn(0x7fb3cc077620 0x7fb3cc079ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:25:56.382 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:56.357+0000 7fb3dcff9640 1 -- 172.21.15.142:0/3753771315 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 6130+0+0 (secure 0 0 0) 0x7fb3d00babf0 con 0x7fb3e01019c0 2024-09-06T14:25:56.383 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:56.357+0000 7fb3deffd640 1 --2- 172.21.15.142:0/3753771315 >> v2:172.21.15.142:6800/2944545612 conn(0x7fb3cc077620 0x7fb3cc079ae0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb3c80095f0 tx=0x7fb3c8005f90 comp rx=0 tx=0).ready entity=mgr.14601 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:25:56.383 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:56.361+0000 7fb3dcff9640 1 -- 172.21.15.142:0/3753771315 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7fb3d0083dc0 con 0x7fb3e01019c0 2024-09-06T14:25:56.558 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:56.561+0000 7fb3e5e19640 1 -- 172.21.15.142:0/3753771315 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fb3e0109b30 con 0x7fb3e01019c0 2024-09-06T14:25:56.558 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:56.561+0000 7fb3dcff9640 1 -- 172.21.15.142:0/3753771315 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+212 (secure 0 0 0) 0x7fb3d0087f30 con 0x7fb3e01019c0 2024-09-06T14:25:56.558 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:25:56.559 INFO:teuthology.orchestra.run.smithi142.stdout:{"status":"HEALTH_WARN","checks":{"MGR_MODULE_DEPENDENCY":{"severity":"HEALTH_WARN","summary":{"message":"Module 'volumes' has failed dependency: No module named 'ceph.fs'","count":1},"muted":false}},"mutes":[]} 2024-09-06T14:25:56.560 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:56.563+0000 7fb3e5e19640 1 -- 172.21.15.142:0/3753771315 >> v2:172.21.15.142:6800/2944545612 conn(0x7fb3cc077620 msgr2=0x7fb3cc079ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:25:56.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:56.563+0000 7fb3e5e19640 1 --2- 172.21.15.142:0/3753771315 >> v2:172.21.15.142:6800/2944545612 conn(0x7fb3cc077620 0x7fb3cc079ae0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb3c80095f0 tx=0x7fb3c8005f90 comp rx=0 tx=0).stop 2024-09-06T14:25:56.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:56.563+0000 7fb3e5e19640 1 -- 172.21.15.142:0/3753771315 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb3e01019c0 msgr2=0x7fb3e010f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:25:56.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:56.563+0000 7fb3e5e19640 1 --2- 172.21.15.142:0/3753771315 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb3e01019c0 0x7fb3e010f060 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fb3d0006010 tx=0x7fb3d00093d0 comp rx=0 tx=0).stop 2024-09-06T14:25:56.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:56.563+0000 7fb3e5e19640 1 -- 172.21.15.142:0/3753771315 shutdown_connections 2024-09-06T14:25:56.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:56.564+0000 7fb3e5e19640 1 --2- 172.21.15.142:0/3753771315 >> v2:172.21.15.142:6800/2944545612 conn(0x7fb3cc077620 0x7fb3cc079ae0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:56.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:56.564+0000 7fb3e5e19640 1 --2- 172.21.15.142:0/3753771315 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb3e0107d50 0x7fb3e010fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:56.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:56.564+0000 7fb3e5e19640 1 --2- 172.21.15.142:0/3753771315 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb3e01073d0 0x7fb3e010f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:56.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:56.564+0000 7fb3e5e19640 1 --2- 172.21.15.142:0/3753771315 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb3e01019c0 0x7fb3e010f060 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:25:56.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:56.564+0000 7fb3e5e19640 1 -- 172.21.15.142:0/3753771315 >> 172.21.15.142:0/3753771315 conn(0x7fb3e00fd430 msgr2=0x7fb3e00ff820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:25:56.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:56.564+0000 7fb3e5e19640 1 -- 172.21.15.142:0/3753771315 shutdown_connections 2024-09-06T14:25:56.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:25:56.564+0000 7fb3e5e19640 1 -- 172.21.15.142:0/3753771315 wait complete. 2024-09-06T14:25:57.128 INFO:journalctl@ceph.alertmanager.a.smithi142.stdout:Sep 06 14:25:56 smithi142 systemd[1]: ceph-ce32462e-6c5a-11ef-bce4-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-09-06T14:25:57.128 INFO:journalctl@ceph.alertmanager.a.smithi142.stdout:Sep 06 14:25:56 smithi142 systemd[1]: Stopped Ceph alertmanager.a for ce32462e-6c5a-11ef-bce4-c7b262605968. 2024-09-06T14:25:57.128 INFO:journalctl@ceph.alertmanager.a.smithi142.stdout:Sep 06 14:25:56 smithi142 systemd[1]: Starting Ceph alertmanager.a for ce32462e-6c5a-11ef-bce4-c7b262605968... 2024-09-06T14:25:57.128 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:56 smithi142 ceph-mon[29958]: pgmap v33: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:25:57.129 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:56 smithi142 ceph-mon[29958]: from='mgr.14601 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]': finished 2024-09-06T14:25:57.129 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:56 smithi142 ceph-mon[29958]: from='mgr.14601 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]': finished 2024-09-06T14:25:57.129 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:56 smithi142 ceph-mon[29958]: from='mgr.14601 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]': finished 2024-09-06T14:25:57.129 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:56 smithi142 ceph-mon[29958]: osdmap e63: 8 total, 8 up, 8 in 2024-09-06T14:25:57.129 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:56 smithi142 ceph-mon[29958]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-06T14:25:57.129 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:56 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/3753771315' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:25:57.129 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:56 smithi142 ceph-mon[37372]: pgmap v33: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:25:57.129 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:56 smithi142 ceph-mon[37372]: from='mgr.14601 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]': finished 2024-09-06T14:25:57.129 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:56 smithi142 ceph-mon[37372]: from='mgr.14601 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]': finished 2024-09-06T14:25:57.130 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:56 smithi142 ceph-mon[37372]: from='mgr.14601 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]': finished 2024-09-06T14:25:57.130 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:56 smithi142 ceph-mon[37372]: osdmap e63: 8 total, 8 up, 8 in 2024-09-06T14:25:57.130 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:56 smithi142 ceph-mon[37372]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-06T14:25:57.130 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:56 smithi142 ceph-mon[37372]: from='client.? 172.21.15.142:0/3753771315' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:25:57.297 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 06 14:25:57 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-iscsi-iscsi-a[59853]: debug there is no tcmu-runner data available 2024-09-06T14:25:57.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:56 smithi162 ceph-mon[33787]: pgmap v33: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:25:57.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:56 smithi162 ceph-mon[33787]: from='mgr.14601 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]': finished 2024-09-06T14:25:57.298 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:56 smithi162 ceph-mon[33787]: from='mgr.14601 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]': finished 2024-09-06T14:25:57.298 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:56 smithi162 ceph-mon[33787]: from='mgr.14601 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]': finished 2024-09-06T14:25:57.298 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:56 smithi162 ceph-mon[33787]: osdmap e63: 8 total, 8 up, 8 in 2024-09-06T14:25:57.298 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:56 smithi162 ceph-mon[33787]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-06T14:25:57.298 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:56 smithi162 ceph-mon[33787]: from='client.? 172.21.15.142:0/3753771315' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:25:57.709 INFO:journalctl@ceph.alertmanager.a.smithi142.stdout:Sep 06 14:25:57 smithi142 podman[72751]: 2024-09-06 14:25:57.337571239 +0000 UTC m=+0.016858793 image pull 11f11916f8cdfb17c1da74b20deb2075d778357b4d60474d82453303f8d461f5 quay.io/prometheus/alertmanager:v0.27.0 2024-09-06T14:25:57.709 INFO:journalctl@ceph.alertmanager.a.smithi142.stdout:Sep 06 14:25:57 smithi142 podman[72751]: 2024-09-06 14:25:57.438294267 +0000 UTC m=+0.117581857 volume create c31a8ef4c7cd4b820c2aba27fd61001f6f18f3acdd99e69289271034856ef7ca 2024-09-06T14:25:57.709 INFO:journalctl@ceph.alertmanager.a.smithi142.stdout:Sep 06 14:25:57 smithi142 podman[72751]: 2024-09-06 14:25:57.555756372 +0000 UTC m=+0.235043961 container create d4eed5ed6c6f478dc7588d07f643a91a333462ad630ac643956587fe5c7618e8 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-06T14:25:58.208 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:57 smithi142 ceph-mon[29958]: osdmap e64: 8 total, 8 up, 8 in 2024-09-06T14:25:58.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:57 smithi142 ceph-mon[37372]: osdmap e64: 8 total, 8 up, 8 in 2024-09-06T14:25:58.209 INFO:journalctl@ceph.alertmanager.a.smithi142.stdout:Sep 06 14:25:57 smithi142 podman[72751]: 2024-09-06 14:25:57.900410464 +0000 UTC m=+0.579698011 container init d4eed5ed6c6f478dc7588d07f643a91a333462ad630ac643956587fe5c7618e8 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-06T14:25:58.209 INFO:journalctl@ceph.alertmanager.a.smithi142.stdout:Sep 06 14:25:57 smithi142 podman[72751]: 2024-09-06 14:25:57.903422608 +0000 UTC m=+0.582710156 container start d4eed5ed6c6f478dc7588d07f643a91a333462ad630ac643956587fe5c7618e8 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-06T14:25:58.209 INFO:journalctl@ceph.alertmanager.a.smithi142.stdout:Sep 06 14:25:57 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-alertmanager-a[72766]: ts=2024-09-06T14:25:57.913Z caller=main.go:181 level=info msg="Starting Alertmanager" version="(version=0.27.0, branch=HEAD, revision=0aa3c2aad14cff039931923ab16b26b7481783b5)" 2024-09-06T14:25:58.209 INFO:journalctl@ceph.alertmanager.a.smithi142.stdout:Sep 06 14:25:57 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-alertmanager-a[72766]: ts=2024-09-06T14:25:57.913Z caller=main.go:182 level=info build_context="(go=go1.21.7, platform=linux/amd64, user=root@22cd11f671e9, date=20240228-11:51:20, tags=netgo)" 2024-09-06T14:25:58.209 INFO:journalctl@ceph.alertmanager.a.smithi142.stdout:Sep 06 14:25:57 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-alertmanager-a[72766]: ts=2024-09-06T14:25:57.914Z caller=cluster.go:186 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.142 port=9094 2024-09-06T14:25:58.210 INFO:journalctl@ceph.alertmanager.a.smithi142.stdout:Sep 06 14:25:57 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-alertmanager-a[72766]: ts=2024-09-06T14:25:57.915Z caller=cluster.go:683 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-09-06T14:25:58.210 INFO:journalctl@ceph.alertmanager.a.smithi142.stdout:Sep 06 14:25:57 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-alertmanager-a[72766]: ts=2024-09-06T14:25:57.937Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-06T14:25:58.210 INFO:journalctl@ceph.alertmanager.a.smithi142.stdout:Sep 06 14:25:57 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-alertmanager-a[72766]: ts=2024-09-06T14:25:57.937Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-06T14:25:58.210 INFO:journalctl@ceph.alertmanager.a.smithi142.stdout:Sep 06 14:25:57 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-alertmanager-a[72766]: ts=2024-09-06T14:25:57.939Z caller=tls_config.go:313 level=info msg="Listening on" address=[::]:9093 2024-09-06T14:25:58.210 INFO:journalctl@ceph.alertmanager.a.smithi142.stdout:Sep 06 14:25:57 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-alertmanager-a[72766]: ts=2024-09-06T14:25:57.939Z caller=tls_config.go:316 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-09-06T14:25:58.210 INFO:journalctl@ceph.alertmanager.a.smithi142.stdout:Sep 06 14:25:58 smithi142 bash[72751]: d4eed5ed6c6f478dc7588d07f643a91a333462ad630ac643956587fe5c7618e8 2024-09-06T14:25:58.210 INFO:journalctl@ceph.alertmanager.a.smithi142.stdout:Sep 06 14:25:58 smithi142 systemd[1]: Started Ceph alertmanager.a for ce32462e-6c5a-11ef-bce4-c7b262605968. 2024-09-06T14:25:58.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:57 smithi162 ceph-mon[33787]: osdmap e64: 8 total, 8 up, 8 in 2024-09-06T14:25:58.972 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:58 smithi162 ceph-mon[33787]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-06T14:25:58.972 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:58 smithi162 ceph-mon[33787]: pgmap v36: 132 pgs: 3 peering, 129 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-06T14:25:58.972 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:58 smithi162 ceph-mon[33787]: Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) 2024-09-06T14:25:58.972 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:58 smithi162 ceph-mon[33787]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:58.972 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:58 smithi162 ceph-mon[33787]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:58.972 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:58 smithi162 ceph-mon[33787]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-06T14:25:59.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:58 smithi142 ceph-mon[29958]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-06T14:25:59.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:58 smithi142 ceph-mon[29958]: pgmap v36: 132 pgs: 3 peering, 129 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-06T14:25:59.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:58 smithi142 ceph-mon[29958]: Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) 2024-09-06T14:25:59.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:58 smithi142 ceph-mon[29958]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:59.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:58 smithi142 ceph-mon[29958]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:59.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:58 smithi142 ceph-mon[29958]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-06T14:25:59.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:58 smithi142 ceph-mon[37372]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-06T14:25:59.210 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:58 smithi142 ceph-mon[37372]: pgmap v36: 132 pgs: 3 peering, 129 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-06T14:25:59.210 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:58 smithi142 ceph-mon[37372]: Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) 2024-09-06T14:25:59.210 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:58 smithi142 ceph-mon[37372]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:59.210 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:58 smithi142 ceph-mon[37372]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:25:59.210 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:58 smithi142 ceph-mon[37372]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-06T14:26:00.154 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:25:59 smithi162 ceph-mon[33787]: Reconfiguring daemon prometheus.a on smithi162 2024-09-06T14:26:00.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:25:59 smithi142 ceph-mon[29958]: Reconfiguring daemon prometheus.a on smithi162 2024-09-06T14:26:00.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:25:59 smithi142 ceph-mon[37372]: Reconfiguring daemon prometheus.a on smithi162 2024-09-06T14:26:00.209 INFO:journalctl@ceph.alertmanager.a.smithi142.stdout:Sep 06 14:25:59 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-alertmanager-a[72766]: ts=2024-09-06T14:25:59.916Z caller=cluster.go:708 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000841785s 2024-09-06T14:26:00.245 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph health --format=json 2024-09-06T14:26:00.511 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:26:00.938 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:26:00 smithi142 ceph-mon[29958]: pgmap v37: 132 pgs: 3 peering, 129 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-09-06T14:26:00.939 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:26:00 smithi142 ceph-mon[37372]: pgmap v37: 132 pgs: 3 peering, 129 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-09-06T14:26:00.966 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:00.941+0000 7f2705e7e640 1 -- 172.21.15.142:0/4028197524 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2700102ae0 msgr2=0x7f2700102ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:26:00.967 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:00.941+0000 7f2705e7e640 1 --2- 172.21.15.142:0/4028197524 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2700102ae0 0x7f2700102ec0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f26f0009900 tx=0x7f26f002a410 comp rx=0 tx=0).stop 2024-09-06T14:26:00.967 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:00.941+0000 7f2705e7e640 1 -- 172.21.15.142:0/4028197524 shutdown_connections 2024-09-06T14:26:00.967 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:00.941+0000 7f2705e7e640 1 --2- 172.21.15.142:0/4028197524 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2700108e70 0x7f270010f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:00.967 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:00.941+0000 7f2705e7e640 1 --2- 172.21.15.142:0/4028197524 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f27001084f0 0x7f2700108930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:00.967 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:00.941+0000 7f2705e7e640 1 --2- 172.21.15.142:0/4028197524 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2700102ae0 0x7f2700102ec0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:00.967 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:00.941+0000 7f2705e7e640 1 -- 172.21.15.142:0/4028197524 >> 172.21.15.142:0/4028197524 conn(0x7f27000fe550 msgr2=0x7f2700100970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:26:00.967 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:00.941+0000 7f2705e7e640 1 -- 172.21.15.142:0/4028197524 shutdown_connections 2024-09-06T14:26:00.967 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:00.941+0000 7f2705e7e640 1 -- 172.21.15.142:0/4028197524 wait complete. 2024-09-06T14:26:00.967 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:00.942+0000 7f2705e7e640 1 Processor -- start 2024-09-06T14:26:00.967 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:00.942+0000 7f2705e7e640 1 -- start start 2024-09-06T14:26:00.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:00.942+0000 7f2705e7e640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2700102ae0 0x7f27001987d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:26:00.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:00.942+0000 7f2705e7e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f27001084f0 0x7f2700198d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:26:00.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:00.942+0000 7f2705e7e640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2700108e70 0x7f270019e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:26:00.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:00.942+0000 7f2705e7e640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f2700112240 con 0x7f2700108e70 2024-09-06T14:26:00.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:00.942+0000 7f2705e7e640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f27001120c0 con 0x7f27001084f0 2024-09-06T14:26:00.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:00.942+0000 7f2705e7e640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7f27001123c0 con 0x7f2700102ae0 2024-09-06T14:26:00.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:00.942+0000 7f26fffff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2700108e70 0x7f270019e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:26:00.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:00.942+0000 7f26ff7fe640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2700102ae0 0x7f27001987d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:26:00.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:00.942+0000 7f26fffff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2700108e70 0x7f270019e0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:45780/0 (socket says 172.21.15.142:45780) 2024-09-06T14:26:00.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:00.942+0000 7f26fffff640 1 -- 172.21.15.142:0/3646465545 learned_addr learned my addr 172.21.15.142:0/3646465545 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:26:00.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:00.942+0000 7f26f6ffd640 1 --2- 172.21.15.142:0/3646465545 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f27001084f0 0x7f2700198d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:26:00.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:00.942+0000 7f26fffff640 1 -- 172.21.15.142:0/3646465545 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2700102ae0 msgr2=0x7f27001987d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:26:00.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:00.942+0000 7f26fffff640 1 --2- 172.21.15.142:0/3646465545 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2700102ae0 0x7f27001987d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:00.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:00.943+0000 7f26fffff640 1 -- 172.21.15.142:0/3646465545 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f27001084f0 msgr2=0x7f2700198d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:26:00.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:00.943+0000 7f26fffff640 1 --2- 172.21.15.142:0/3646465545 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f27001084f0 0x7f2700198d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:00.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:00.943+0000 7f26fffff640 1 -- 172.21.15.142:0/3646465545 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f270019e880 con 0x7f2700108e70 2024-09-06T14:26:00.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:00.943+0000 7f26f6ffd640 1 --2- 172.21.15.142:0/3646465545 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f27001084f0 0x7f2700198d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-06T14:26:00.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:00.943+0000 7f26fffff640 1 --2- 172.21.15.142:0/3646465545 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2700108e70 0x7f270019e0b0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f26f800efd0 tx=0x7f26f800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:26:00.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:00.943+0000 7f26fd7fa640 1 -- 172.21.15.142:0/3646465545 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26f8003b70 con 0x7f2700108e70 2024-09-06T14:26:00.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:00.943+0000 7f2705e7e640 1 -- 172.21.15.142:0/3646465545 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f270019eb70 con 0x7f2700108e70 2024-09-06T14:26:00.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:00.943+0000 7f26fd7fa640 1 -- 172.21.15.142:0/3646465545 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7f26f8003d10 con 0x7f2700108e70 2024-09-06T14:26:00.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:00.943+0000 7f26fd7fa640 1 -- 172.21.15.142:0/3646465545 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26f8010840 con 0x7f2700108e70 2024-09-06T14:26:00.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:00.943+0000 7f2705e7e640 1 -- 172.21.15.142:0/3646465545 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27000750e0 con 0x7f2700108e70 2024-09-06T14:26:00.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:00.944+0000 7f2705e7e640 1 -- 172.21.15.142:0/3646465545 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2700102ec0 con 0x7f2700108e70 2024-09-06T14:26:00.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:00.944+0000 7f26fd7fa640 1 -- 172.21.15.142:0/3646465545 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 22) ==== 99844+0+0 (secure 0 0 0) 0x7f26f8004020 con 0x7f2700108e70 2024-09-06T14:26:00.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:00.945+0000 7f26fd7fa640 1 --2- 172.21.15.142:0/3646465545 >> v2:172.21.15.142:6800/2944545612 conn(0x7f26ec077620 0x7f26ec079ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:26:00.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:00.945+0000 7f26ff7fe640 1 --2- 172.21.15.142:0/3646465545 >> v2:172.21.15.142:6800/2944545612 conn(0x7f26ec077620 0x7f26ec079ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:26:00.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:00.945+0000 7f26fd7fa640 1 -- 172.21.15.142:0/3646465545 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 6130+0+0 (secure 0 0 0) 0x7f26f8099e00 con 0x7f2700108e70 2024-09-06T14:26:00.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:00.946+0000 7f26ff7fe640 1 --2- 172.21.15.142:0/3646465545 >> v2:172.21.15.142:6800/2944545612 conn(0x7f26ec077620 0x7f26ec079ae0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f26f0031440 tx=0x7f26f003c040 comp rx=0 tx=0).ready entity=mgr.14601 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:26:00.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:00.949+0000 7f26fd7fa640 1 -- 172.21.15.142:0/3646465545 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7f26f8062ec0 con 0x7f2700108e70 2024-09-06T14:26:01.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:26:00 smithi162 ceph-mon[33787]: pgmap v37: 132 pgs: 3 peering, 129 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-09-06T14:26:01.138 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:01.140+0000 7f2705e7e640 1 -- 172.21.15.142:0/3646465545 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f270010ac50 con 0x7f2700108e70 2024-09-06T14:26:01.138 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:01.141+0000 7f26fd7fa640 1 -- 172.21.15.142:0/3646465545 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+346 (secure 0 0 0) 0x7f26f8067030 con 0x7f2700108e70 2024-09-06T14:26:01.138 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:26:01.138 INFO:teuthology.orchestra.run.smithi142.stdout:{"status":"HEALTH_WARN","checks":{"MGR_MODULE_DEPENDENCY":{"severity":"HEALTH_WARN","summary":{"message":"Module 'volumes' has failed dependency: No module named 'ceph.fs'","count":1},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 3 pgs peering","count":3},"muted":false}},"mutes":[]} 2024-09-06T14:26:01.140 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:01.143+0000 7f2705e7e640 1 -- 172.21.15.142:0/3646465545 >> v2:172.21.15.142:6800/2944545612 conn(0x7f26ec077620 msgr2=0x7f26ec079ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:26:01.140 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:01.143+0000 7f2705e7e640 1 --2- 172.21.15.142:0/3646465545 >> v2:172.21.15.142:6800/2944545612 conn(0x7f26ec077620 0x7f26ec079ae0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f26f0031440 tx=0x7f26f003c040 comp rx=0 tx=0).stop 2024-09-06T14:26:01.140 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:01.143+0000 7f2705e7e640 1 -- 172.21.15.142:0/3646465545 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2700108e70 msgr2=0x7f270019e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:26:01.140 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:01.143+0000 7f2705e7e640 1 --2- 172.21.15.142:0/3646465545 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2700108e70 0x7f270019e0b0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f26f800efd0 tx=0x7f26f800c6a0 comp rx=0 tx=0).stop 2024-09-06T14:26:01.140 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:01.143+0000 7f2705e7e640 1 -- 172.21.15.142:0/3646465545 shutdown_connections 2024-09-06T14:26:01.140 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:01.143+0000 7f2705e7e640 1 --2- 172.21.15.142:0/3646465545 >> v2:172.21.15.142:6800/2944545612 conn(0x7f26ec077620 0x7f26ec079ae0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:01.140 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:01.143+0000 7f2705e7e640 1 --2- 172.21.15.142:0/3646465545 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2700108e70 0x7f270019e0b0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:01.140 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:01.143+0000 7f2705e7e640 1 --2- 172.21.15.142:0/3646465545 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f27001084f0 0x7f2700198d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:01.140 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:01.143+0000 7f2705e7e640 1 --2- 172.21.15.142:0/3646465545 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2700102ae0 0x7f27001987d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:01.140 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:01.143+0000 7f2705e7e640 1 -- 172.21.15.142:0/3646465545 >> 172.21.15.142:0/3646465545 conn(0x7f27000fe550 msgr2=0x7f2700100940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:26:01.140 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:01.143+0000 7f2705e7e640 1 -- 172.21.15.142:0/3646465545 shutdown_connections 2024-09-06T14:26:01.141 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:01.143+0000 7f2705e7e640 1 -- 172.21.15.142:0/3646465545 wait complete. 2024-09-06T14:26:01.797 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 06 14:26:01 smithi162 systemd[1]: Stopping Ceph prometheus.a for ce32462e-6c5a-11ef-bce4-c7b262605968... 2024-09-06T14:26:01.897 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:26:01 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/3646465545' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:26:01.898 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:26:01 smithi142 ceph-mon[37372]: from='client.? 172.21.15.142:0/3646465545' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:26:02.153 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:26:01 smithi162 ceph-mon[33787]: from='client.? 172.21.15.142:0/3646465545' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:26:02.153 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 06 14:26:01 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-prometheus-a[60775]: ts=2024-09-06T14:26:01.872Z caller=main.go:964 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-06T14:26:02.153 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 06 14:26:01 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-prometheus-a[60775]: ts=2024-09-06T14:26:01.873Z caller=main.go:988 level=info msg="Stopping scrape discovery manager..." 2024-09-06T14:26:02.153 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 06 14:26:01 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-prometheus-a[60775]: ts=2024-09-06T14:26:01.873Z caller=main.go:1002 level=info msg="Stopping notify discovery manager..." 2024-09-06T14:26:02.153 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 06 14:26:01 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-prometheus-a[60775]: ts=2024-09-06T14:26:01.873Z caller=manager.go:177 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-06T14:26:02.153 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 06 14:26:01 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-prometheus-a[60775]: ts=2024-09-06T14:26:01.873Z caller=main.go:984 level=info msg="Scrape discovery manager stopped" 2024-09-06T14:26:02.153 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 06 14:26:01 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-prometheus-a[60775]: ts=2024-09-06T14:26:01.873Z caller=main.go:998 level=info msg="Notify discovery manager stopped" 2024-09-06T14:26:02.154 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 06 14:26:01 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-prometheus-a[60775]: ts=2024-09-06T14:26:01.873Z caller=manager.go:187 level=info component="rule manager" msg="Rule manager stopped" 2024-09-06T14:26:02.154 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 06 14:26:01 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-prometheus-a[60775]: ts=2024-09-06T14:26:01.873Z caller=main.go:1039 level=info msg="Stopping scrape manager..." 2024-09-06T14:26:02.154 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 06 14:26:01 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-prometheus-a[60775]: ts=2024-09-06T14:26:01.873Z caller=main.go:1031 level=info msg="Scrape manager stopped" 2024-09-06T14:26:02.154 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 06 14:26:01 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-prometheus-a[60775]: ts=2024-09-06T14:26:01.873Z caller=notifier.go:618 level=info component=notifier msg="Stopping notification manager..." 2024-09-06T14:26:02.154 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 06 14:26:01 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-prometheus-a[60775]: ts=2024-09-06T14:26:01.874Z caller=main.go:1261 level=info msg="Notifier manager stopped" 2024-09-06T14:26:02.154 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 06 14:26:01 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-prometheus-a[60775]: ts=2024-09-06T14:26:01.874Z caller=main.go:1273 level=info msg="See you next time!" 2024-09-06T14:26:02.154 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 06 14:26:01 smithi162 podman[63898]: 2024-09-06 14:26:01.883245649 +0000 UTC m=+0.147632688 container died fe7a84ccb7effa272383f6bc6b7ccb526f1c4ce50c78335a7dd93350af707e78 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-06T14:26:02.503 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 06 14:26:02 smithi162 podman[63898]: 2024-09-06 14:26:02.255663016 +0000 UTC m=+0.520050063 container cleanup fe7a84ccb7effa272383f6bc6b7ccb526f1c4ce50c78335a7dd93350af707e78 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-06T14:26:02.503 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 06 14:26:02 smithi162 bash[63898]: ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-prometheus-a 2024-09-06T14:26:02.797 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 06 14:26:02 smithi162 podman[63910]: 2024-09-06 14:26:02.507296927 +0000 UTC m=+0.621956150 container remove fe7a84ccb7effa272383f6bc6b7ccb526f1c4ce50c78335a7dd93350af707e78 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-06T14:26:02.797 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 06 14:26:02 smithi162 systemd[1]: ceph-ce32462e-6c5a-11ef-bce4-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-09-06T14:26:02.797 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 06 14:26:02 smithi162 systemd[1]: Stopped Ceph prometheus.a for ce32462e-6c5a-11ef-bce4-c7b262605968. 2024-09-06T14:26:02.798 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 06 14:26:02 smithi162 systemd[1]: Starting Ceph prometheus.a for ce32462e-6c5a-11ef-bce4-c7b262605968... 2024-09-06T14:26:03.140 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 06 14:26:02 smithi162 podman[64000]: 2024-09-06 14:26:02.799292651 +0000 UTC m=+0.017000779 image pull 1d3b7f56885b6dd623f1785be963aa9c195f86bc256ea454e8d02a7980b79c53 quay.io/prometheus/prometheus:v2.51.0 2024-09-06T14:26:03.140 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 06 14:26:02 smithi162 podman[64000]: 2024-09-06 14:26:02.924951578 +0000 UTC m=+0.142659699 container create f5d9f09518b211980b023ad3cd696941462f57fa427a957b5eea62b9e75ca41e (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-06T14:26:03.140 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:26:02 smithi162 ceph-mon[33787]: pgmap v38: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 54 B/s, 1 objects/s recovering 2024-09-06T14:26:03.140 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:26:02 smithi162 ceph-mon[33787]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering) 2024-09-06T14:26:03.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:26:02 smithi142 ceph-mon[29958]: pgmap v38: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 54 B/s, 1 objects/s recovering 2024-09-06T14:26:03.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:26:02 smithi142 ceph-mon[29958]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering) 2024-09-06T14:26:03.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:26:02 smithi142 ceph-mon[37372]: pgmap v38: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 54 B/s, 1 objects/s recovering 2024-09-06T14:26:03.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:26:02 smithi142 ceph-mon[37372]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering) 2024-09-06T14:26:03.548 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 06 14:26:03 smithi162 podman[64000]: 2024-09-06 14:26:03.143884745 +0000 UTC m=+0.361592855 container init f5d9f09518b211980b023ad3cd696941462f57fa427a957b5eea62b9e75ca41e (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-06T14:26:03.548 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 06 14:26:03 smithi162 podman[64000]: 2024-09-06 14:26:03.146898087 +0000 UTC m=+0.364606210 container start f5d9f09518b211980b023ad3cd696941462f57fa427a957b5eea62b9e75ca41e (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-06T14:26:03.548 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 06 14:26:03 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-prometheus-a[64015]: ts=2024-09-06T14:26:03.166Z caller=main.go:617 level=info msg="Starting Prometheus Server" mode=server version="(version=2.51.0, branch=HEAD, revision=c05c15512acb675e3f6cd662a6727854e93fc024)" 2024-09-06T14:26:03.548 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 06 14:26:03 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-prometheus-a[64015]: ts=2024-09-06T14:26:03.166Z caller=main.go:622 level=info build_context="(go=go1.22.1, platform=linux/amd64, user=root@b5723e458358, date=20240319-10:54:45, tags=netgo,builtinassets,stringlabels)" 2024-09-06T14:26:03.548 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 06 14:26:03 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-prometheus-a[64015]: ts=2024-09-06T14:26:03.166Z caller=main.go:623 level=info host_details="(Linux 5.14.0-503.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Aug 22 17:03:23 UTC 2024 x86_64 smithi162 (none))" 2024-09-06T14:26:03.548 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 06 14:26:03 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-prometheus-a[64015]: ts=2024-09-06T14:26:03.166Z caller=main.go:624 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-09-06T14:26:03.548 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 06 14:26:03 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-prometheus-a[64015]: ts=2024-09-06T14:26:03.167Z caller=main.go:625 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-06T14:26:03.548 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 06 14:26:03 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-prometheus-a[64015]: ts=2024-09-06T14:26:03.168Z caller=web.go:568 level=info component=web msg="Start listening for connections" address=:9095 2024-09-06T14:26:03.548 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 06 14:26:03 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-prometheus-a[64015]: ts=2024-09-06T14:26:03.169Z caller=main.go:1129 level=info msg="Starting TSDB ..." 2024-09-06T14:26:03.548 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 06 14:26:03 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-prometheus-a[64015]: ts=2024-09-06T14:26:03.171Z caller=tls_config.go:313 level=info component=web msg="Listening on" address=[::]:9095 2024-09-06T14:26:03.549 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 06 14:26:03 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-prometheus-a[64015]: ts=2024-09-06T14:26:03.171Z caller=tls_config.go:316 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-09-06T14:26:03.549 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 06 14:26:03 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-prometheus-a[64015]: ts=2024-09-06T14:26:03.173Z caller=head.go:616 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-09-06T14:26:03.549 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 06 14:26:03 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-prometheus-a[64015]: ts=2024-09-06T14:26:03.173Z caller=head.go:698 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.609µs 2024-09-06T14:26:03.549 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 06 14:26:03 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-prometheus-a[64015]: ts=2024-09-06T14:26:03.173Z caller=head.go:706 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-09-06T14:26:03.549 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 06 14:26:03 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-prometheus-a[64015]: ts=2024-09-06T14:26:03.173Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-09-06T14:26:03.549 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 06 14:26:03 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-prometheus-a[64015]: ts=2024-09-06T14:26:03.173Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-09-06T14:26:03.549 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 06 14:26:03 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-prometheus-a[64015]: ts=2024-09-06T14:26:03.173Z caller=head.go:815 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=32.948µs wal_replay_duration=401.86µs wbl_replay_duration=113ns total_replay_duration=453.293µs 2024-09-06T14:26:03.549 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 06 14:26:03 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-prometheus-a[64015]: ts=2024-09-06T14:26:03.175Z caller=main.go:1150 level=info fs_type=XFS_SUPER_MAGIC 2024-09-06T14:26:03.549 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 06 14:26:03 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-prometheus-a[64015]: ts=2024-09-06T14:26:03.175Z caller=main.go:1153 level=info msg="TSDB started" 2024-09-06T14:26:03.549 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 06 14:26:03 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-prometheus-a[64015]: ts=2024-09-06T14:26:03.175Z caller=main.go:1335 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-06T14:26:03.549 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 06 14:26:03 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-prometheus-a[64015]: ts=2024-09-06T14:26:03.191Z caller=main.go:1372 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=15.637768ms db_storage=914ns remote_storage=1.17µs web_handler=387ns query_engine=529ns scrape=378.102µs scrape_sd=96.805µs notify=10.074µs notify_sd=9.04µs rules=14.810927ms tracing=5.202µs 2024-09-06T14:26:03.549 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 06 14:26:03 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-prometheus-a[64015]: ts=2024-09-06T14:26:03.191Z caller=main.go:1114 level=info msg="Server is ready to receive web requests." 2024-09-06T14:26:03.550 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 06 14:26:03 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-prometheus-a[64015]: ts=2024-09-06T14:26:03.191Z caller=manager.go:163 level=info component="rule manager" msg="Starting rule manager..." 2024-09-06T14:26:03.550 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 06 14:26:03 smithi162 bash[64000]: f5d9f09518b211980b023ad3cd696941462f57fa427a957b5eea62b9e75ca41e 2024-09-06T14:26:03.550 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 06 14:26:03 smithi162 systemd[1]: Started Ceph prometheus.a for ce32462e-6c5a-11ef-bce4-c7b262605968. 2024-09-06T14:26:03.711 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:26:03 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: [06/Sep/2024:14:26:03] ENGINE Bus STOPPING 2024-09-06T14:26:03.712 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:26:03 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: [06/Sep/2024:14:26:03] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-06T14:26:03.712 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:26:03 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: [06/Sep/2024:14:26:03] ENGINE Bus STOPPED 2024-09-06T14:26:03.712 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:26:03 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: [06/Sep/2024:14:26:03] ENGINE Bus STARTING 2024-09-06T14:26:03.712 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:26:03 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: [06/Sep/2024:14:26:03] ENGINE Serving on http://:::9283 2024-09-06T14:26:03.712 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:26:03 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: [06/Sep/2024:14:26:03] ENGINE Bus STARTED 2024-09-06T14:26:03.712 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:26:03 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: [06/Sep/2024:14:26:03] ENGINE Bus STOPPING 2024-09-06T14:26:04.248 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:26:03 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: [06/Sep/2024:14:26:03] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-06T14:26:04.248 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:26:03 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: [06/Sep/2024:14:26:03] ENGINE Bus STOPPED 2024-09-06T14:26:04.248 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:26:03 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: [06/Sep/2024:14:26:03] ENGINE Bus STARTING 2024-09-06T14:26:04.248 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:26:04 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: [06/Sep/2024:14:26:04] ENGINE Serving on http://:::9283 2024-09-06T14:26:04.249 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:26:04 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: [06/Sep/2024:14:26:04] ENGINE Bus STARTED 2024-09-06T14:26:04.249 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:26:04 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: [06/Sep/2024:14:26:04] ENGINE Bus STOPPING 2024-09-06T14:26:04.547 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:26:04 smithi162 ceph-mon[33787]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:26:04.547 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:26:04 smithi162 ceph-mon[33787]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:26:04.547 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:26:04 smithi162 ceph-mon[33787]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "dashboard get-alertmanager-api-host"} : dispatch 2024-09-06T14:26:04.548 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:26:04 smithi162 ceph-mon[33787]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-06T14:26:04.548 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:26:04 smithi162 ceph-mon[33787]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi142:9093"} : dispatch 2024-09-06T14:26:04.548 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:26:04 smithi162 ceph-mon[33787]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi142:9093"}]: dispatch 2024-09-06T14:26:04.548 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:26:04 smithi162 ceph-mon[33787]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:26:04.548 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:26:04 smithi162 ceph-mon[33787]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "dashboard get-prometheus-api-host"} : dispatch 2024-09-06T14:26:04.548 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:26:04 smithi162 ceph-mon[33787]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-06T14:26:04.548 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:26:04 smithi162 ceph-mon[33787]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi162:9095"} : dispatch 2024-09-06T14:26:04.548 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:26:04 smithi162 ceph-mon[33787]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi162:9095"}]: dispatch 2024-09-06T14:26:04.548 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:26:04 smithi162 ceph-mon[33787]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:26:04.548 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:26:04 smithi162 ceph-mon[33787]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "dashboard get-grafana-api-url"} : dispatch 2024-09-06T14:26:04.548 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:26:04 smithi162 ceph-mon[33787]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-06T14:26:04.548 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:26:04 smithi162 ceph-mon[33787]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "dashboard set-grafana-api-url", "value": "https://smithi162:3000"} : dispatch 2024-09-06T14:26:04.548 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:26:04 smithi162 ceph-mon[33787]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi162:3000"}]: dispatch 2024-09-06T14:26:04.549 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:26:04 smithi162 ceph-mon[33787]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:26:04.549 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:26:04 smithi162 ceph-mon[33787]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-06T14:26:04.549 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:26:04 smithi162 ceph-mon[33787]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-06T14:26:04.549 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:26:04 smithi162 ceph-mon[33787]: pgmap v39: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 54 B/s, 1 objects/s recovering 2024-09-06T14:26:04.549 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:26:04 smithi162 ceph-mon[33787]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-06T14:26:04.549 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:26:04 smithi162 ceph-mon[33787]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-06T14:26:04.549 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:26:04 smithi162 ceph-mon[33787]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-06T14:26:04.592 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:26:04 smithi142 ceph-mon[37372]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:26:04.592 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:26:04 smithi142 ceph-mon[37372]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:26:04.592 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:26:04 smithi142 ceph-mon[37372]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "dashboard get-alertmanager-api-host"} : dispatch 2024-09-06T14:26:04.592 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:26:04 smithi142 ceph-mon[37372]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-06T14:26:04.592 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:26:04 smithi142 ceph-mon[37372]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi142:9093"} : dispatch 2024-09-06T14:26:04.592 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:26:04 smithi142 ceph-mon[37372]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi142:9093"}]: dispatch 2024-09-06T14:26:04.593 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:26:04 smithi142 ceph-mon[37372]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:26:04.593 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:26:04 smithi142 ceph-mon[37372]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "dashboard get-prometheus-api-host"} : dispatch 2024-09-06T14:26:04.593 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:26:04 smithi142 ceph-mon[37372]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-06T14:26:04.593 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:26:04 smithi142 ceph-mon[37372]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi162:9095"} : dispatch 2024-09-06T14:26:04.593 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:26:04 smithi142 ceph-mon[37372]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi162:9095"}]: dispatch 2024-09-06T14:26:04.593 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:26:04 smithi142 ceph-mon[37372]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:26:04.593 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:26:04 smithi142 ceph-mon[37372]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "dashboard get-grafana-api-url"} : dispatch 2024-09-06T14:26:04.593 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:26:04 smithi142 ceph-mon[37372]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-06T14:26:04.593 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:26:04 smithi142 ceph-mon[37372]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "dashboard set-grafana-api-url", "value": "https://smithi162:3000"} : dispatch 2024-09-06T14:26:04.593 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:26:04 smithi142 ceph-mon[37372]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi162:3000"}]: dispatch 2024-09-06T14:26:04.593 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:26:04 smithi142 ceph-mon[37372]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:26:04.593 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:26:04 smithi142 ceph-mon[37372]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-06T14:26:04.593 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:26:04 smithi142 ceph-mon[37372]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-06T14:26:04.594 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:26:04 smithi142 ceph-mon[37372]: pgmap v39: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 54 B/s, 1 objects/s recovering 2024-09-06T14:26:04.594 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:26:04 smithi142 ceph-mon[37372]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-06T14:26:04.594 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:26:04 smithi142 ceph-mon[37372]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-06T14:26:04.594 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:26:04 smithi142 ceph-mon[37372]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-06T14:26:04.594 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:26:04 smithi142 ceph-mon[29958]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:26:04.594 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:26:04 smithi142 ceph-mon[29958]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:26:04.594 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:26:04 smithi142 ceph-mon[29958]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "dashboard get-alertmanager-api-host"} : dispatch 2024-09-06T14:26:04.594 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:26:04 smithi142 ceph-mon[29958]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-06T14:26:04.594 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:26:04 smithi142 ceph-mon[29958]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi142:9093"} : dispatch 2024-09-06T14:26:04.594 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:26:04 smithi142 ceph-mon[29958]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi142:9093"}]: dispatch 2024-09-06T14:26:04.594 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:26:04 smithi142 ceph-mon[29958]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:26:04.595 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:26:04 smithi142 ceph-mon[29958]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "dashboard get-prometheus-api-host"} : dispatch 2024-09-06T14:26:04.595 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:26:04 smithi142 ceph-mon[29958]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-06T14:26:04.595 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:26:04 smithi142 ceph-mon[29958]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi162:9095"} : dispatch 2024-09-06T14:26:04.595 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:26:04 smithi142 ceph-mon[29958]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi162:9095"}]: dispatch 2024-09-06T14:26:04.595 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:26:04 smithi142 ceph-mon[29958]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:26:04.595 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:26:04 smithi142 ceph-mon[29958]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "dashboard get-grafana-api-url"} : dispatch 2024-09-06T14:26:04.595 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:26:04 smithi142 ceph-mon[29958]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-06T14:26:04.595 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:26:04 smithi142 ceph-mon[29958]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "dashboard set-grafana-api-url", "value": "https://smithi162:3000"} : dispatch 2024-09-06T14:26:04.595 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:26:04 smithi142 ceph-mon[29958]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi162:3000"}]: dispatch 2024-09-06T14:26:04.595 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:26:04 smithi142 ceph-mon[29958]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:26:04.595 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:26:04 smithi142 ceph-mon[29958]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-06T14:26:04.595 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:26:04 smithi142 ceph-mon[29958]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-06T14:26:04.595 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:26:04 smithi142 ceph-mon[29958]: pgmap v39: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 54 B/s, 1 objects/s recovering 2024-09-06T14:26:04.595 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:26:04 smithi142 ceph-mon[29958]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-06T14:26:04.595 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:26:04 smithi142 ceph-mon[29958]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-06T14:26:04.596 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:26:04 smithi142 ceph-mon[29958]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-06T14:26:04.880 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:26:04 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: [06/Sep/2024:14:26:04] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-06T14:26:04.880 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:26:04 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: [06/Sep/2024:14:26:04] ENGINE Bus STOPPED 2024-09-06T14:26:04.881 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:26:04 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: [06/Sep/2024:14:26:04] ENGINE Bus STARTING 2024-09-06T14:26:04.881 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:26:04 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: [06/Sep/2024:14:26:04] ENGINE Serving on http://:::9283 2024-09-06T14:26:04.881 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:26:04 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: [06/Sep/2024:14:26:04] ENGINE Bus STARTED 2024-09-06T14:26:04.899 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph health --format=json 2024-09-06T14:26:05.459 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:26:05 smithi142 ceph-mon[29958]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-06T14:26:05.459 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:26:05 smithi142 ceph-mon[29958]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-06T14:26:05.459 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:26:05 smithi142 ceph-mon[37372]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-06T14:26:05.459 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:26:05 smithi142 ceph-mon[37372]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-06T14:26:05.547 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:26:05 smithi162 ceph-mon[33787]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-06T14:26:05.547 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:26:05 smithi162 ceph-mon[33787]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-06T14:26:05.648 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:26:06.652 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:26:06 smithi162 ceph-mon[33787]: pgmap v40: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 951 B/s rd, 0 op/s; 45 B/s, 1 objects/s recovering 2024-09-06T14:26:06.693 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:26:06 smithi142 ceph-mon[29958]: pgmap v40: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 951 B/s rd, 0 op/s; 45 B/s, 1 objects/s recovering 2024-09-06T14:26:06.694 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:26:06 smithi142 ceph-mon[37372]: pgmap v40: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 951 B/s rd, 0 op/s; 45 B/s, 1 objects/s recovering 2024-09-06T14:26:06.778 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:06.776+0000 7f39b90a9640 1 -- 172.21.15.142:0/2416410801 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f39b4102ae0 msgr2=0x7f39b4102ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:26:06.778 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:06.776+0000 7f39b90a9640 1 --2- 172.21.15.142:0/2416410801 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f39b4102ae0 0x7f39b4102ec0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f39a4009860 tx=0x7f39a402a340 comp rx=0 tx=0).stop 2024-09-06T14:26:06.778 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:06.776+0000 7f39b90a9640 1 -- 172.21.15.142:0/2416410801 shutdown_connections 2024-09-06T14:26:06.778 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:06.776+0000 7f39b90a9640 1 --2- 172.21.15.142:0/2416410801 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f39b4108e70 0x7f39b410f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:06.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:06.776+0000 7f39b90a9640 1 --2- 172.21.15.142:0/2416410801 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f39b41084f0 0x7f39b4108930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:06.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:06.776+0000 7f39b90a9640 1 --2- 172.21.15.142:0/2416410801 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f39b4102ae0 0x7f39b4102ec0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:06.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:06.776+0000 7f39b90a9640 1 -- 172.21.15.142:0/2416410801 >> 172.21.15.142:0/2416410801 conn(0x7f39b40fe530 msgr2=0x7f39b4100950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:26:06.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:06.776+0000 7f39b90a9640 1 -- 172.21.15.142:0/2416410801 shutdown_connections 2024-09-06T14:26:06.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:06.776+0000 7f39b90a9640 1 -- 172.21.15.142:0/2416410801 wait complete. 2024-09-06T14:26:06.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:06.777+0000 7f39b90a9640 1 Processor -- start 2024-09-06T14:26:06.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:06.777+0000 7f39b90a9640 1 -- start start 2024-09-06T14:26:06.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:06.777+0000 7f39b90a9640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f39b4102ae0 0x7f39b4075200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:26:06.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:06.777+0000 7f39b90a9640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f39b41084f0 0x7f39b4075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:26:06.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:06.777+0000 7f39b90a9640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f39b4108e70 0x7f39b4075c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:26:06.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:06.777+0000 7f39b90a9640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f39b4112240 con 0x7f39b41084f0 2024-09-06T14:26:06.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:06.777+0000 7f39b90a9640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f39b41120c0 con 0x7f39b4102ae0 2024-09-06T14:26:06.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:06.777+0000 7f39b90a9640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7f39b41123c0 con 0x7f39b4108e70 2024-09-06T14:26:06.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:06.777+0000 7f39b359e640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f39b4108e70 0x7f39b4075c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:26:06.780 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:06.777+0000 7f39b2d9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f39b4102ae0 0x7f39b4075200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:26:06.780 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:06.777+0000 7f39b359e640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f39b4108e70 0x7f39b4075c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:59766/0 (socket says 172.21.15.142:59766) 2024-09-06T14:26:06.780 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:06.777+0000 7f39b359e640 1 -- 172.21.15.142:0/2768342858 learned_addr learned my addr 172.21.15.142:0/2768342858 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:26:06.780 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:06.777+0000 7f39b2d9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f39b4102ae0 0x7f39b4075200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.142:50962/0 (socket says 172.21.15.142:50962) 2024-09-06T14:26:06.780 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:06.777+0000 7f39b259c640 1 --2- 172.21.15.142:0/2768342858 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f39b41084f0 0x7f39b4075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:26:06.780 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:06.778+0000 7f39b359e640 1 -- 172.21.15.142:0/2768342858 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f39b4102ae0 msgr2=0x7f39b4075200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:26:06.780 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:06.778+0000 7f39b359e640 1 --2- 172.21.15.142:0/2768342858 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f39b4102ae0 0x7f39b4075200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:06.780 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:06.778+0000 7f39b359e640 1 -- 172.21.15.142:0/2768342858 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f39b41084f0 msgr2=0x7f39b4075740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:26:06.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:06.778+0000 7f39b359e640 1 --2- 172.21.15.142:0/2768342858 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f39b41084f0 0x7f39b4075740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:06.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:06.778+0000 7f39b359e640 1 -- 172.21.15.142:0/2768342858 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39b41a6bd0 con 0x7f39b4108e70 2024-09-06T14:26:06.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:06.778+0000 7f39b259c640 1 --2- 172.21.15.142:0/2768342858 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f39b41084f0 0x7f39b4075740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-06T14:26:06.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:06.778+0000 7f39b2d9d640 1 --2- 172.21.15.142:0/2768342858 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f39b4102ae0 0x7f39b4075200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-06T14:26:06.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:06.778+0000 7f39b359e640 1 --2- 172.21.15.142:0/2768342858 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f39b4108e70 0x7f39b4075c80 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f39ac00efc0 tx=0x7f39ac00c800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:26:06.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:06.778+0000 7f39abfff640 1 -- 172.21.15.142:0/2768342858 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39ac003970 con 0x7f39b4108e70 2024-09-06T14:26:06.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:06.778+0000 7f39abfff640 1 -- 172.21.15.142:0/2768342858 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7f39ac004020 con 0x7f39b4108e70 2024-09-06T14:26:06.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:06.778+0000 7f39b90a9640 1 -- 172.21.15.142:0/2768342858 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39b41a6ec0 con 0x7f39b4108e70 2024-09-06T14:26:06.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:06.778+0000 7f39abfff640 1 -- 172.21.15.142:0/2768342858 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39ac00b830 con 0x7f39b4108e70 2024-09-06T14:26:06.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:06.778+0000 7f39b90a9640 1 -- 172.21.15.142:0/2768342858 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f39b41a7320 con 0x7f39b4108e70 2024-09-06T14:26:06.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:06.779+0000 7f39b90a9640 1 -- 172.21.15.142:0/2768342858 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39b4102ec0 con 0x7f39b4108e70 2024-09-06T14:26:06.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:06.780+0000 7f39abfff640 1 -- 172.21.15.142:0/2768342858 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 22) ==== 99844+0+0 (secure 0 0 0) 0x7f39ac01d020 con 0x7f39b4108e70 2024-09-06T14:26:06.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:06.780+0000 7f39abfff640 1 --2- 172.21.15.142:0/2768342858 >> v2:172.21.15.142:6800/2944545612 conn(0x7f39a0077620 0x7f39a0079ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:26:06.782 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:06.780+0000 7f39abfff640 1 -- 172.21.15.142:0/2768342858 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(64..64 src has 1..64) ==== 6130+0+0 (secure 0 0 0) 0x7f39ac0996a0 con 0x7f39b4108e70 2024-09-06T14:26:06.782 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:06.780+0000 7f39b2d9d640 1 --2- 172.21.15.142:0/2768342858 >> v2:172.21.15.142:6800/2944545612 conn(0x7f39a0077620 0x7f39a0079ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:26:06.782 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:06.781+0000 7f39b2d9d640 1 --2- 172.21.15.142:0/2768342858 >> v2:172.21.15.142:6800/2944545612 conn(0x7f39a0077620 0x7f39a0079ae0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f39a402a930 tx=0x7f39a40056d0 comp rx=0 tx=0).ready entity=mgr.14601 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:26:06.782 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:06.784+0000 7f39abfff640 1 -- 172.21.15.142:0/2768342858 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7f39ac0668d0 con 0x7f39b4108e70 2024-09-06T14:26:06.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:06.972+0000 7f39b90a9640 1 -- 172.21.15.142:0/2768342858 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f39b410ac50 con 0x7f39b4108e70 2024-09-06T14:26:06.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:06.972+0000 7f39abfff640 1 -- 172.21.15.142:0/2768342858 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+212 (secure 0 0 0) 0x7f39ac06b2f0 con 0x7f39b4108e70 2024-09-06T14:26:06.969 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:26:06.969 INFO:teuthology.orchestra.run.smithi142.stdout:{"status":"HEALTH_WARN","checks":{"MGR_MODULE_DEPENDENCY":{"severity":"HEALTH_WARN","summary":{"message":"Module 'volumes' has failed dependency: No module named 'ceph.fs'","count":1},"muted":false}},"mutes":[]} 2024-09-06T14:26:06.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:06.974+0000 7f39b90a9640 1 -- 172.21.15.142:0/2768342858 >> v2:172.21.15.142:6800/2944545612 conn(0x7f39a0077620 msgr2=0x7f39a0079ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:26:06.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:06.974+0000 7f39b90a9640 1 --2- 172.21.15.142:0/2768342858 >> v2:172.21.15.142:6800/2944545612 conn(0x7f39a0077620 0x7f39a0079ae0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f39a402a930 tx=0x7f39a40056d0 comp rx=0 tx=0).stop 2024-09-06T14:26:06.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:06.974+0000 7f39b90a9640 1 -- 172.21.15.142:0/2768342858 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f39b4108e70 msgr2=0x7f39b4075c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:26:06.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:06.974+0000 7f39b90a9640 1 --2- 172.21.15.142:0/2768342858 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f39b4108e70 0x7f39b4075c80 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f39ac00efc0 tx=0x7f39ac00c800 comp rx=0 tx=0).stop 2024-09-06T14:26:06.972 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:06.974+0000 7f39b90a9640 1 -- 172.21.15.142:0/2768342858 shutdown_connections 2024-09-06T14:26:06.972 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:06.974+0000 7f39b90a9640 1 --2- 172.21.15.142:0/2768342858 >> v2:172.21.15.142:6800/2944545612 conn(0x7f39a0077620 0x7f39a0079ae0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:06.972 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:06.974+0000 7f39b90a9640 1 --2- 172.21.15.142:0/2768342858 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f39b4108e70 0x7f39b4075c80 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:06.972 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:06.974+0000 7f39b90a9640 1 --2- 172.21.15.142:0/2768342858 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f39b41084f0 0x7f39b4075740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:06.972 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:06.974+0000 7f39b90a9640 1 --2- 172.21.15.142:0/2768342858 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f39b4102ae0 0x7f39b4075200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:06.972 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:06.974+0000 7f39b90a9640 1 -- 172.21.15.142:0/2768342858 >> 172.21.15.142:0/2768342858 conn(0x7f39b40fe530 msgr2=0x7f39b40fff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:26:06.972 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:06.974+0000 7f39b90a9640 1 -- 172.21.15.142:0/2768342858 shutdown_connections 2024-09-06T14:26:06.972 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:06.974+0000 7f39b90a9640 1 -- 172.21.15.142:0/2768342858 wait complete. 2024-09-06T14:26:07.358 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 06 14:26:07 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-iscsi-iscsi-a[59853]: debug there is no tcmu-runner data available 2024-09-06T14:26:07.797 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:26:07 smithi162 ceph-mon[33787]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:26:07.797 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:26:07 smithi162 ceph-mon[33787]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:26:07.797 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:26:07 smithi162 ceph-mon[33787]: from='client.? 172.21.15.142:0/2768342858' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:26:07.915 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:26:07 smithi142 ceph-mon[29958]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:26:07.915 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:26:07 smithi142 ceph-mon[29958]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:26:07.916 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:26:07 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/2768342858' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:26:07.916 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:26:07 smithi142 ceph-mon[37372]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:26:07.916 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:26:07 smithi142 ceph-mon[37372]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:26:07.916 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:26:07 smithi142 ceph-mon[37372]: from='client.? 172.21.15.142:0/2768342858' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:26:08.209 INFO:journalctl@ceph.alertmanager.a.smithi142.stdout:Sep 06 14:26:07 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-alertmanager-a[72766]: ts=2024-09-06T14:26:07.918Z caller=cluster.go:700 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.003424058s 2024-09-06T14:26:08.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:26:08 smithi142 ceph-mon[29958]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-06T14:26:08.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:26:08 smithi142 ceph-mon[29958]: pgmap v41: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 958 B/s rd, 0 op/s; 41 B/s, 1 objects/s recovering 2024-09-06T14:26:08.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:26:08 smithi142 ceph-mon[29958]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:26:08.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:26:08 smithi142 ceph-mon[29958]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:26:08.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:26:08 smithi142 ceph-mon[29958]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:26:08.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:26:08 smithi142 ceph-mon[29958]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-06T14:26:08.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:26:08 smithi142 ceph-mon[29958]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:26:08.959 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:26:08 smithi142 ceph-mon[37372]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-06T14:26:08.960 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:26:08 smithi142 ceph-mon[37372]: pgmap v41: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 958 B/s rd, 0 op/s; 41 B/s, 1 objects/s recovering 2024-09-06T14:26:08.960 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:26:08 smithi142 ceph-mon[37372]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:26:08.960 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:26:08 smithi142 ceph-mon[37372]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:26:08.960 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:26:08 smithi142 ceph-mon[37372]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:26:08.960 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:26:08 smithi142 ceph-mon[37372]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-06T14:26:08.960 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:26:08 smithi142 ceph-mon[37372]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:26:09.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:26:08 smithi162 ceph-mon[33787]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-06T14:26:09.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:26:08 smithi162 ceph-mon[33787]: pgmap v41: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 958 B/s rd, 0 op/s; 41 B/s, 1 objects/s recovering 2024-09-06T14:26:09.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:26:08 smithi162 ceph-mon[33787]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:26:09.048 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:26:08 smithi162 ceph-mon[33787]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:26:09.048 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:26:08 smithi162 ceph-mon[33787]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:26:09.048 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:26:08 smithi162 ceph-mon[33787]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-06T14:26:09.048 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:26:08 smithi162 ceph-mon[33787]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:26:10.720 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph health --format=json 2024-09-06T14:26:10.958 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:26:10 smithi142 ceph-mon[29958]: pgmap v42: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 36 B/s, 0 objects/s recovering 2024-09-06T14:26:10.959 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:26:10 smithi142 ceph-mon[37372]: pgmap v42: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 36 B/s, 0 objects/s recovering 2024-09-06T14:26:10.981 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:26:11.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:26:10 smithi162 ceph-mon[33787]: pgmap v42: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 36 B/s, 0 objects/s recovering 2024-09-06T14:26:11.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:11.501+0000 7febfb83b640 1 -- 172.21.15.142:0/2152099273 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7febf41073d0 msgr2=0x7febf4107810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:26:11.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:11.501+0000 7febfb83b640 1 --2- 172.21.15.142:0/2152099273 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7febf41073d0 0x7febf4107810 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7febe8009900 tx=0x7febe802a410 comp rx=0 tx=0).stop 2024-09-06T14:26:11.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:11.502+0000 7febfb83b640 1 -- 172.21.15.142:0/2152099273 shutdown_connections 2024-09-06T14:26:11.588 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:11.502+0000 7febfb83b640 1 --2- 172.21.15.142:0/2152099273 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7febf4107d50 0x7febf410e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:11.588 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:11.502+0000 7febfb83b640 1 --2- 172.21.15.142:0/2152099273 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7febf41073d0 0x7febf4107810 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:11.588 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:11.502+0000 7febfb83b640 1 --2- 172.21.15.142:0/2152099273 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7febf41019c0 0x7febf4101da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:11.588 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:11.502+0000 7febfb83b640 1 -- 172.21.15.142:0/2152099273 >> 172.21.15.142:0/2152099273 conn(0x7febf40fd430 msgr2=0x7febf40ff850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:26:11.588 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:11.502+0000 7febfb83b640 1 -- 172.21.15.142:0/2152099273 shutdown_connections 2024-09-06T14:26:11.588 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:11.502+0000 7febfb83b640 1 -- 172.21.15.142:0/2152099273 wait complete. 2024-09-06T14:26:11.588 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:11.502+0000 7febfb83b640 1 Processor -- start 2024-09-06T14:26:11.588 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:11.502+0000 7febfb83b640 1 -- start start 2024-09-06T14:26:11.588 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:11.502+0000 7febfb83b640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7febf41019c0 0x7febf4198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:26:11.588 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:11.502+0000 7febfb83b640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7febf41073d0 0x7febf4198c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:26:11.588 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:11.502+0000 7febfb83b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7febf4107d50 0x7febf419dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:26:11.588 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:11.502+0000 7febfb83b640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7febf4111170 con 0x7febf41073d0 2024-09-06T14:26:11.589 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:11.502+0000 7febfb83b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7febf4110ff0 con 0x7febf4107d50 2024-09-06T14:26:11.589 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:11.502+0000 7febfb83b640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7febf41112f0 con 0x7febf41019c0 2024-09-06T14:26:11.589 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:11.503+0000 7febf95d7640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7febf41019c0 0x7febf4198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:26:11.589 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:11.503+0000 7febf8dd6640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7febf41073d0 0x7febf4198c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:26:11.589 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:11.503+0000 7febf95d7640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7febf41019c0 0x7febf4198700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:49370/0 (socket says 172.21.15.142:49370) 2024-09-06T14:26:11.589 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:11.503+0000 7febf95d7640 1 -- 172.21.15.142:0/2704118352 learned_addr learned my addr 172.21.15.142:0/2704118352 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:26:11.589 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:11.503+0000 7febf95d7640 1 -- 172.21.15.142:0/2704118352 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7febf4107d50 msgr2=0x7febf419dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:26:11.589 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:11.503+0000 7febf95d7640 1 --2- 172.21.15.142:0/2704118352 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7febf4107d50 0x7febf419dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:11.589 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:11.503+0000 7febf95d7640 1 -- 172.21.15.142:0/2704118352 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7febf41073d0 msgr2=0x7febf4198c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:26:11.589 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:11.503+0000 7febf95d7640 1 --2- 172.21.15.142:0/2704118352 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7febf41073d0 0x7febf4198c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:11.589 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:11.503+0000 7febf95d7640 1 -- 172.21.15.142:0/2704118352 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febf419e7b0 con 0x7febf41019c0 2024-09-06T14:26:11.589 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:11.503+0000 7febf8dd6640 1 --2- 172.21.15.142:0/2704118352 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7febf41073d0 0x7febf4198c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-06T14:26:11.590 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:11.503+0000 7febf95d7640 1 --2- 172.21.15.142:0/2704118352 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7febf41019c0 0x7febf4198700 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7febf000d770 tx=0x7febf000dc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:26:11.590 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:11.503+0000 7febe67fc640 1 -- 172.21.15.142:0/2704118352 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febf0017070 con 0x7febf41019c0 2024-09-06T14:26:11.590 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:11.503+0000 7febe67fc640 1 -- 172.21.15.142:0/2704118352 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7febf00054a0 con 0x7febf41019c0 2024-09-06T14:26:11.590 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:11.503+0000 7febfb83b640 1 -- 172.21.15.142:0/2704118352 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7febf419eaa0 con 0x7febf41019c0 2024-09-06T14:26:11.590 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:11.503+0000 7febe67fc640 1 -- 172.21.15.142:0/2704118352 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febf0004c40 con 0x7febf41019c0 2024-09-06T14:26:11.590 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:11.503+0000 7febfb83b640 1 -- 172.21.15.142:0/2704118352 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7febf41a5160 con 0x7febf41019c0 2024-09-06T14:26:11.590 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:11.504+0000 7febfb83b640 1 -- 172.21.15.142:0/2704118352 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7febf4101da0 con 0x7febf41019c0 2024-09-06T14:26:11.590 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:11.505+0000 7febe67fc640 1 -- 172.21.15.142:0/2704118352 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 22) ==== 99844+0+0 (secure 0 0 0) 0x7febf0004de0 con 0x7febf41019c0 2024-09-06T14:26:11.590 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:11.505+0000 7febe67fc640 1 --2- 172.21.15.142:0/2704118352 >> v2:172.21.15.142:6800/2944545612 conn(0x7febe0077620 0x7febe0079ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:26:11.590 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:11.506+0000 7febf8dd6640 1 --2- 172.21.15.142:0/2704118352 >> v2:172.21.15.142:6800/2944545612 conn(0x7febe0077620 0x7febe0079ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:26:11.590 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:11.506+0000 7febe67fc640 1 -- 172.21.15.142:0/2704118352 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(64..64 src has 1..64) ==== 6130+0+0 (secure 0 0 0) 0x7febf00993d0 con 0x7febf41019c0 2024-09-06T14:26:11.590 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:11.506+0000 7febf8dd6640 1 --2- 172.21.15.142:0/2704118352 >> v2:172.21.15.142:6800/2944545612 conn(0x7febe0077620 0x7febe0079ae0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7febe8005ab0 tx=0x7febe8005a20 comp rx=0 tx=0).ready entity=mgr.14601 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:26:11.591 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:11.510+0000 7febe67fc640 1 -- 172.21.15.142:0/2704118352 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7febf0066680 con 0x7febf41019c0 2024-09-06T14:26:11.699 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:11.702+0000 7febfb83b640 1 -- 172.21.15.142:0/2704118352 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7febf410ab40 con 0x7febf41019c0 2024-09-06T14:26:11.703 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:11.706+0000 7febe67fc640 1 -- 172.21.15.142:0/2704118352 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+212 (secure 0 0 0) 0x7febf006b0a0 con 0x7febf41019c0 2024-09-06T14:26:11.703 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:26:11.703 INFO:teuthology.orchestra.run.smithi142.stdout:{"status":"HEALTH_WARN","checks":{"MGR_MODULE_DEPENDENCY":{"severity":"HEALTH_WARN","summary":{"message":"Module 'volumes' has failed dependency: No module named 'ceph.fs'","count":1},"muted":false}},"mutes":[]} 2024-09-06T14:26:11.705 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:11.708+0000 7febfb83b640 1 -- 172.21.15.142:0/2704118352 >> v2:172.21.15.142:6800/2944545612 conn(0x7febe0077620 msgr2=0x7febe0079ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:26:11.706 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:11.708+0000 7febfb83b640 1 --2- 172.21.15.142:0/2704118352 >> v2:172.21.15.142:6800/2944545612 conn(0x7febe0077620 0x7febe0079ae0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7febe8005ab0 tx=0x7febe8005a20 comp rx=0 tx=0).stop 2024-09-06T14:26:11.706 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:11.708+0000 7febfb83b640 1 -- 172.21.15.142:0/2704118352 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7febf41019c0 msgr2=0x7febf4198700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:26:11.706 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:11.708+0000 7febfb83b640 1 --2- 172.21.15.142:0/2704118352 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7febf41019c0 0x7febf4198700 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7febf000d770 tx=0x7febf000dc30 comp rx=0 tx=0).stop 2024-09-06T14:26:11.706 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:11.708+0000 7febfb83b640 1 -- 172.21.15.142:0/2704118352 shutdown_connections 2024-09-06T14:26:11.706 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:11.708+0000 7febfb83b640 1 --2- 172.21.15.142:0/2704118352 >> v2:172.21.15.142:6800/2944545612 conn(0x7febe0077620 0x7febe0079ae0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:11.706 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:11.708+0000 7febfb83b640 1 --2- 172.21.15.142:0/2704118352 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7febf4107d50 0x7febf419dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:11.706 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:11.708+0000 7febfb83b640 1 --2- 172.21.15.142:0/2704118352 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7febf41073d0 0x7febf4198c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:11.706 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:11.708+0000 7febfb83b640 1 --2- 172.21.15.142:0/2704118352 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7febf41019c0 0x7febf4198700 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:11.706 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:11.708+0000 7febfb83b640 1 -- 172.21.15.142:0/2704118352 >> 172.21.15.142:0/2704118352 conn(0x7febf40fd430 msgr2=0x7febf406aa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:26:11.706 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:11.708+0000 7febfb83b640 1 -- 172.21.15.142:0/2704118352 shutdown_connections 2024-09-06T14:26:11.706 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:11.708+0000 7febfb83b640 1 -- 172.21.15.142:0/2704118352 wait complete. 2024-09-06T14:26:11.959 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:26:11 smithi142 ceph-mon[37372]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-06T14:26:11.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:26:11 smithi142 ceph-mon[29958]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-06T14:26:12.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:26:11 smithi162 ceph-mon[33787]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-06T14:26:12.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:26:12 smithi142 ceph-mon[29958]: pgmap v43: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 36 B/s, 0 objects/s recovering 2024-09-06T14:26:12.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:26:12 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/2704118352' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:26:12.959 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:26:12 smithi142 ceph-mon[37372]: pgmap v43: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 36 B/s, 0 objects/s recovering 2024-09-06T14:26:12.959 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:26:12 smithi142 ceph-mon[37372]: from='client.? 172.21.15.142:0/2704118352' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:26:13.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:26:12 smithi162 ceph-mon[33787]: pgmap v43: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 36 B/s, 0 objects/s recovering 2024-09-06T14:26:13.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:26:12 smithi162 ceph-mon[33787]: from='client.? 172.21.15.142:0/2704118352' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:26:14.704 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:26:14 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: ::ffff:172.21.15.162 - - [06/Sep/2024:14:26:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-06T14:26:14.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:26:14 smithi142 ceph-mon[29958]: pgmap v44: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:26:14.959 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:26:14 smithi142 ceph-mon[37372]: pgmap v44: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:26:15.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:26:14 smithi162 ceph-mon[33787]: pgmap v44: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:26:15.472 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph health --format=json 2024-09-06T14:26:15.732 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:26:16.228 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:16.228+0000 7f45937f6640 1 -- 172.21.15.142:0/987788555 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f458c1019c0 msgr2=0x7f458c101da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:26:16.228 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:16.228+0000 7f45937f6640 1 --2- 172.21.15.142:0/987788555 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f458c1019c0 0x7f458c101da0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4588009860 tx=0x7f458802a340 comp rx=0 tx=0).stop 2024-09-06T14:26:16.228 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:16.228+0000 7f45937f6640 1 -- 172.21.15.142:0/987788555 shutdown_connections 2024-09-06T14:26:16.228 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:16.228+0000 7f45937f6640 1 --2- 172.21.15.142:0/987788555 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f458c107d50 0x7f458c10e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:16.228 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:16.228+0000 7f45937f6640 1 --2- 172.21.15.142:0/987788555 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f458c1073d0 0x7f458c107810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:16.228 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:16.228+0000 7f45937f6640 1 --2- 172.21.15.142:0/987788555 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f458c1019c0 0x7f458c101da0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:16.228 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:16.228+0000 7f45937f6640 1 -- 172.21.15.142:0/987788555 >> 172.21.15.142:0/987788555 conn(0x7f458c0fd430 msgr2=0x7f458c0ff850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:26:16.228 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:16.229+0000 7f45937f6640 1 -- 172.21.15.142:0/987788555 shutdown_connections 2024-09-06T14:26:16.228 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:16.229+0000 7f45937f6640 1 -- 172.21.15.142:0/987788555 wait complete. 2024-09-06T14:26:16.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:16.229+0000 7f45937f6640 1 Processor -- start 2024-09-06T14:26:16.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:16.229+0000 7f45937f6640 1 -- start start 2024-09-06T14:26:16.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:16.229+0000 7f45937f6640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f458c1019c0 0x7f458c10eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:26:16.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:16.229+0000 7f45937f6640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f458c1073d0 0x7f458c10f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:26:16.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:16.229+0000 7f45937f6640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f458c107d50 0x7f458c10f930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:26:16.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:16.229+0000 7f45937f6640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f458c112200 con 0x7f458c1073d0 2024-09-06T14:26:16.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:16.229+0000 7f45937f6640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f458c112080 con 0x7f458c1019c0 2024-09-06T14:26:16.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:16.229+0000 7f45937f6640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7f458c112380 con 0x7f458c107d50 2024-09-06T14:26:16.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:16.229+0000 7f4590d91640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f458c1073d0 0x7f458c10f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:26:16.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:16.229+0000 7f4590d91640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f458c1073d0 0x7f458c10f3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:46398/0 (socket says 172.21.15.142:46398) 2024-09-06T14:26:16.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:16.229+0000 7f4590d91640 1 -- 172.21.15.142:0/3448291769 learned_addr learned my addr 172.21.15.142:0/3448291769 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:26:16.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:16.229+0000 7f4591d93640 1 --2- 172.21.15.142:0/3448291769 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f458c107d50 0x7f458c10f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:26:16.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:16.230+0000 7f4591592640 1 --2- 172.21.15.142:0/3448291769 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f458c1019c0 0x7f458c10eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:26:16.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:16.230+0000 7f4590d91640 1 -- 172.21.15.142:0/3448291769 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f458c107d50 msgr2=0x7f458c10f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:26:16.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:16.230+0000 7f4590d91640 1 --2- 172.21.15.142:0/3448291769 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f458c107d50 0x7f458c10f930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:16.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:16.230+0000 7f4590d91640 1 -- 172.21.15.142:0/3448291769 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f458c1019c0 msgr2=0x7f458c10eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:26:16.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:16.230+0000 7f4590d91640 1 --2- 172.21.15.142:0/3448291769 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f458c1019c0 0x7f458c10eeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:16.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:16.230+0000 7f4590d91640 1 -- 172.21.15.142:0/3448291769 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f458c1a6b70 con 0x7f458c1073d0 2024-09-06T14:26:16.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:16.230+0000 7f4590d91640 1 --2- 172.21.15.142:0/3448291769 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f458c1073d0 0x7f458c10f3f0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f457c00c970 tx=0x7f457c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:26:16.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:16.230+0000 7f45827fc640 1 -- 172.21.15.142:0/3448291769 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f457c007bb0 con 0x7f458c1073d0 2024-09-06T14:26:16.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:16.230+0000 7f45937f6640 1 -- 172.21.15.142:0/3448291769 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f458c1a6e60 con 0x7f458c1073d0 2024-09-06T14:26:16.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:16.230+0000 7f45827fc640 1 -- 172.21.15.142:0/3448291769 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7f457c007d50 con 0x7f458c1073d0 2024-09-06T14:26:16.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:16.230+0000 7f45827fc640 1 -- 172.21.15.142:0/3448291769 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f457c004da0 con 0x7f458c1073d0 2024-09-06T14:26:16.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:16.230+0000 7f45937f6640 1 -- 172.21.15.142:0/3448291769 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f458c1a7310 con 0x7f458c1073d0 2024-09-06T14:26:16.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:16.231+0000 7f45937f6640 1 -- 172.21.15.142:0/3448291769 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f458c101da0 con 0x7f458c1073d0 2024-09-06T14:26:16.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:16.232+0000 7f45827fc640 1 -- 172.21.15.142:0/3448291769 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 22) ==== 99844+0+0 (secure 0 0 0) 0x7f457c004f80 con 0x7f458c1073d0 2024-09-06T14:26:16.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:16.232+0000 7f45827fc640 1 --2- 172.21.15.142:0/3448291769 >> v2:172.21.15.142:6800/2944545612 conn(0x7f4578077620 0x7f4578079ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:26:16.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:16.233+0000 7f45827fc640 1 -- 172.21.15.142:0/3448291769 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 6130+0+0 (secure 0 0 0) 0x7f457c099630 con 0x7f458c1073d0 2024-09-06T14:26:16.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:16.233+0000 7f4591592640 1 --2- 172.21.15.142:0/3448291769 >> v2:172.21.15.142:6800/2944545612 conn(0x7f4578077620 0x7f4578079ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:26:16.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:16.233+0000 7f4591592640 1 --2- 172.21.15.142:0/3448291769 >> v2:172.21.15.142:6800/2944545612 conn(0x7f4578077620 0x7f4578079ae0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f458802a930 tx=0x7f45880056d0 comp rx=0 tx=0).ready entity=mgr.14601 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:26:16.233 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:16.237+0000 7f45827fc640 1 -- 172.21.15.142:0/3448291769 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7f457c014030 con 0x7f458c1073d0 2024-09-06T14:26:16.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:16.431+0000 7f45937f6640 1 -- 172.21.15.142:0/3448291769 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f458c02e370 con 0x7f458c1073d0 2024-09-06T14:26:16.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:16.432+0000 7f45827fc640 1 -- 172.21.15.142:0/3448291769 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+212 (secure 0 0 0) 0x7f457c066860 con 0x7f458c1073d0 2024-09-06T14:26:16.429 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:26:16.429 INFO:teuthology.orchestra.run.smithi142.stdout:{"status":"HEALTH_WARN","checks":{"MGR_MODULE_DEPENDENCY":{"severity":"HEALTH_WARN","summary":{"message":"Module 'volumes' has failed dependency: No module named 'ceph.fs'","count":1},"muted":false}},"mutes":[]} 2024-09-06T14:26:16.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:16.434+0000 7f45937f6640 1 -- 172.21.15.142:0/3448291769 >> v2:172.21.15.142:6800/2944545612 conn(0x7f4578077620 msgr2=0x7f4578079ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:26:16.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:16.434+0000 7f45937f6640 1 --2- 172.21.15.142:0/3448291769 >> v2:172.21.15.142:6800/2944545612 conn(0x7f4578077620 0x7f4578079ae0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f458802a930 tx=0x7f45880056d0 comp rx=0 tx=0).stop 2024-09-06T14:26:16.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:16.434+0000 7f45937f6640 1 -- 172.21.15.142:0/3448291769 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f458c1073d0 msgr2=0x7f458c10f3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:26:16.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:16.434+0000 7f45937f6640 1 --2- 172.21.15.142:0/3448291769 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f458c1073d0 0x7f458c10f3f0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f457c00c970 tx=0x7f457c00ce30 comp rx=0 tx=0).stop 2024-09-06T14:26:16.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:16.434+0000 7f45937f6640 1 -- 172.21.15.142:0/3448291769 shutdown_connections 2024-09-06T14:26:16.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:16.434+0000 7f45937f6640 1 --2- 172.21.15.142:0/3448291769 >> v2:172.21.15.142:6800/2944545612 conn(0x7f4578077620 0x7f4578079ae0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:16.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:16.434+0000 7f45937f6640 1 --2- 172.21.15.142:0/3448291769 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f458c107d50 0x7f458c10f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:16.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:16.434+0000 7f45937f6640 1 --2- 172.21.15.142:0/3448291769 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f458c1073d0 0x7f458c10f3f0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:16.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:16.434+0000 7f45937f6640 1 --2- 172.21.15.142:0/3448291769 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f458c1019c0 0x7f458c10eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:16.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:16.434+0000 7f45937f6640 1 -- 172.21.15.142:0/3448291769 >> 172.21.15.142:0/3448291769 conn(0x7f458c0fd430 msgr2=0x7f458c0ff820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:26:16.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:16.434+0000 7f45937f6640 1 -- 172.21.15.142:0/3448291769 shutdown_connections 2024-09-06T14:26:16.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:16.434+0000 7f45937f6640 1 -- 172.21.15.142:0/3448291769 wait complete. 2024-09-06T14:26:16.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:26:16 smithi142 ceph-mon[29958]: pgmap v45: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:26:16.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:26:16 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/3448291769' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:26:16.959 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:26:16 smithi142 ceph-mon[37372]: pgmap v45: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:26:16.959 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:26:16 smithi142 ceph-mon[37372]: from='client.? 172.21.15.142:0/3448291769' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:26:17.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:26:16 smithi162 ceph-mon[33787]: pgmap v45: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:26:17.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:26:16 smithi162 ceph-mon[33787]: from='client.? 172.21.15.142:0/3448291769' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:26:17.547 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 06 14:26:17 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-iscsi-iscsi-a[59853]: debug there is no tcmu-runner data available 2024-09-06T14:26:19.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:26:18 smithi162 ceph-mon[33787]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-06T14:26:19.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:26:18 smithi162 ceph-mon[33787]: pgmap v46: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:26:19.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:26:18 smithi142 ceph-mon[29958]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-06T14:26:19.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:26:18 smithi142 ceph-mon[29958]: pgmap v46: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:26:19.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:26:18 smithi142 ceph-mon[37372]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-06T14:26:19.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:26:18 smithi142 ceph-mon[37372]: pgmap v46: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:26:20.211 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph health --format=json 2024-09-06T14:26:20.475 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:26:20.894 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:26:20 smithi142 ceph-mon[29958]: pgmap v47: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:26:20.894 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:26:20 smithi142 ceph-mon[37372]: pgmap v47: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:26:20.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:20.977+0000 7f3841cb0640 1 -- 172.21.15.142:0/1346438662 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f383c107d70 msgr2=0x7f383c10e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:26:20.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:20.977+0000 7f3841cb0640 1 --2- 172.21.15.142:0/1346438662 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f383c107d70 0x7f383c10e570 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f3834009be0 tx=0x7f383402a6c0 comp rx=0 tx=0).stop 2024-09-06T14:26:20.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:20.977+0000 7f3841cb0640 1 -- 172.21.15.142:0/1346438662 shutdown_connections 2024-09-06T14:26:20.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:20.977+0000 7f3841cb0640 1 --2- 172.21.15.142:0/1346438662 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f383c107d70 0x7f383c10e570 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:20.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:20.977+0000 7f3841cb0640 1 --2- 172.21.15.142:0/1346438662 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f383c1073f0 0x7f383c107830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:20.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:20.977+0000 7f3841cb0640 1 --2- 172.21.15.142:0/1346438662 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f383c1019e0 0x7f383c101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:20.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:20.977+0000 7f3841cb0640 1 -- 172.21.15.142:0/1346438662 >> 172.21.15.142:0/1346438662 conn(0x7f383c0fd430 msgr2=0x7f383c0ff850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:26:20.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:20.977+0000 7f3841cb0640 1 -- 172.21.15.142:0/1346438662 shutdown_connections 2024-09-06T14:26:20.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:20.977+0000 7f3841cb0640 1 -- 172.21.15.142:0/1346438662 wait complete. 2024-09-06T14:26:20.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:20.978+0000 7f3841cb0640 1 Processor -- start 2024-09-06T14:26:20.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:20.978+0000 7f3841cb0640 1 -- start start 2024-09-06T14:26:20.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:20.978+0000 7f3841cb0640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f383c1019e0 0x7f383c1994d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:26:20.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:20.978+0000 7f3841cb0640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f383c1073f0 0x7f383c199a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:26:20.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:20.978+0000 7f3841cb0640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f383c107d70 0x7f383c199f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:26:20.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:20.978+0000 7f3841cb0640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f383c111100 con 0x7f383c107d70 2024-09-06T14:26:20.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:20.978+0000 7f3841cb0640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f383c110f80 con 0x7f383c1019e0 2024-09-06T14:26:20.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:20.978+0000 7f3841cb0640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7f383c111280 con 0x7f383c1073f0 2024-09-06T14:26:20.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:20.978+0000 7f3833fff640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f383c1073f0 0x7f383c199a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:26:20.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:20.978+0000 7f3833fff640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f383c1073f0 0x7f383c199a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:46436/0 (socket says 172.21.15.142:46436) 2024-09-06T14:26:20.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:20.978+0000 7f383bfff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f383c107d70 0x7f383c199f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:26:20.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:20.978+0000 7f3833fff640 1 -- 172.21.15.142:0/3660817307 learned_addr learned my addr 172.21.15.142:0/3660817307 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:26:20.988 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:20.978+0000 7f383b7fe640 1 --2- 172.21.15.142:0/3660817307 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f383c1019e0 0x7f383c1994d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:26:20.988 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:20.978+0000 7f383bfff640 1 -- 172.21.15.142:0/3660817307 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f383c1073f0 msgr2=0x7f383c199a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:26:20.988 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:20.978+0000 7f383bfff640 1 --2- 172.21.15.142:0/3660817307 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f383c1073f0 0x7f383c199a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:20.988 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:20.978+0000 7f383bfff640 1 -- 172.21.15.142:0/3660817307 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f383c1019e0 msgr2=0x7f383c1994d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:26:20.988 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:20.978+0000 7f383bfff640 1 --2- 172.21.15.142:0/3660817307 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f383c1019e0 0x7f383c1994d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:20.988 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:20.978+0000 7f383bfff640 1 -- 172.21.15.142:0/3660817307 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f383c1a48b0 con 0x7f383c107d70 2024-09-06T14:26:20.988 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:20.978+0000 7f383bfff640 1 --2- 172.21.15.142:0/3660817307 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f383c107d70 0x7f383c199f70 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f383402abe0 tx=0x7f3834030650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:26:20.988 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:20.978+0000 7f38397fa640 1 -- 172.21.15.142:0/3660817307 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3834041020 con 0x7f383c107d70 2024-09-06T14:26:20.988 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:20.979+0000 7f3841cb0640 1 -- 172.21.15.142:0/3660817307 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f383c1a4b40 con 0x7f383c107d70 2024-09-06T14:26:20.988 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:20.979+0000 7f38397fa640 1 -- 172.21.15.142:0/3660817307 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7f3834003970 con 0x7f383c107d70 2024-09-06T14:26:20.988 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:20.979+0000 7f38397fa640 1 -- 172.21.15.142:0/3660817307 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3834036040 con 0x7f383c107d70 2024-09-06T14:26:20.989 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:20.979+0000 7f3841cb0640 1 -- 172.21.15.142:0/3660817307 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f383c1a4ff0 con 0x7f383c107d70 2024-09-06T14:26:20.989 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:20.980+0000 7f3841cb0640 1 -- 172.21.15.142:0/3660817307 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f383c101dc0 con 0x7f383c107d70 2024-09-06T14:26:20.989 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:20.980+0000 7f38397fa640 1 -- 172.21.15.142:0/3660817307 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 22) ==== 99844+0+0 (secure 0 0 0) 0x7f3834044020 con 0x7f383c107d70 2024-09-06T14:26:20.989 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:20.981+0000 7f38397fa640 1 --2- 172.21.15.142:0/3660817307 >> v2:172.21.15.142:6800/2944545612 conn(0x7f3828077620 0x7f3828079ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:26:20.989 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:20.981+0000 7f383b7fe640 1 --2- 172.21.15.142:0/3660817307 >> v2:172.21.15.142:6800/2944545612 conn(0x7f3828077620 0x7f3828079ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:26:20.989 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:20.981+0000 7f38397fa640 1 -- 172.21.15.142:0/3660817307 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 6130+0+0 (secure 0 0 0) 0x7f38340ba570 con 0x7f383c107d70 2024-09-06T14:26:20.989 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:20.981+0000 7f383b7fe640 1 --2- 172.21.15.142:0/3660817307 >> v2:172.21.15.142:6800/2944545612 conn(0x7f3828077620 0x7f3828079ae0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f382c009530 tx=0x7f382c009260 comp rx=0 tx=0).ready entity=mgr.14601 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:26:20.989 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:20.985+0000 7f38397fa640 1 -- 172.21.15.142:0/3660817307 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7f383404b020 con 0x7f383c107d70 2024-09-06T14:26:21.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:26:20 smithi162 ceph-mon[33787]: pgmap v47: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:26:21.171 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:21.174+0000 7f3841cb0640 1 -- 172.21.15.142:0/3660817307 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f383c1a51c0 con 0x7f383c107d70 2024-09-06T14:26:21.171 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:21.174+0000 7f38397fa640 1 -- 172.21.15.142:0/3660817307 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+212 (secure 0 0 0) 0x7f38340877a0 con 0x7f383c107d70 2024-09-06T14:26:21.171 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:26:21.171 INFO:teuthology.orchestra.run.smithi142.stdout:{"status":"HEALTH_WARN","checks":{"MGR_MODULE_DEPENDENCY":{"severity":"HEALTH_WARN","summary":{"message":"Module 'volumes' has failed dependency: No module named 'ceph.fs'","count":1},"muted":false}},"mutes":[]} 2024-09-06T14:26:21.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:21.176+0000 7f3841cb0640 1 -- 172.21.15.142:0/3660817307 >> v2:172.21.15.142:6800/2944545612 conn(0x7f3828077620 msgr2=0x7f3828079ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:26:21.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:21.176+0000 7f3841cb0640 1 --2- 172.21.15.142:0/3660817307 >> v2:172.21.15.142:6800/2944545612 conn(0x7f3828077620 0x7f3828079ae0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f382c009530 tx=0x7f382c009260 comp rx=0 tx=0).stop 2024-09-06T14:26:21.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:21.176+0000 7f3841cb0640 1 -- 172.21.15.142:0/3660817307 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f383c107d70 msgr2=0x7f383c199f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:26:21.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:21.176+0000 7f3841cb0640 1 --2- 172.21.15.142:0/3660817307 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f383c107d70 0x7f383c199f70 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f383402abe0 tx=0x7f3834030650 comp rx=0 tx=0).stop 2024-09-06T14:26:21.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:21.176+0000 7f3841cb0640 1 -- 172.21.15.142:0/3660817307 shutdown_connections 2024-09-06T14:26:21.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:21.176+0000 7f3841cb0640 1 --2- 172.21.15.142:0/3660817307 >> v2:172.21.15.142:6800/2944545612 conn(0x7f3828077620 0x7f3828079ae0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:21.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:21.176+0000 7f3841cb0640 1 --2- 172.21.15.142:0/3660817307 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f383c107d70 0x7f383c199f70 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:21.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:21.176+0000 7f3841cb0640 1 --2- 172.21.15.142:0/3660817307 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f383c1073f0 0x7f383c199a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:21.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:21.176+0000 7f3841cb0640 1 --2- 172.21.15.142:0/3660817307 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f383c1019e0 0x7f383c1994d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:21.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:21.176+0000 7f3841cb0640 1 -- 172.21.15.142:0/3660817307 >> 172.21.15.142:0/3660817307 conn(0x7f383c0fd430 msgr2=0x7f383c0fec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:26:21.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:21.176+0000 7f3841cb0640 1 -- 172.21.15.142:0/3660817307 shutdown_connections 2024-09-06T14:26:21.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:21.176+0000 7f3841cb0640 1 -- 172.21.15.142:0/3660817307 wait complete. 2024-09-06T14:26:21.848 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:26:21 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/3660817307' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:26:21.848 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:26:21 smithi142 ceph-mon[37372]: from='client.? 172.21.15.142:0/3660817307' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:26:22.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:26:21 smithi162 ceph-mon[33787]: from='client.? 172.21.15.142:0/3660817307' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:26:23.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:26:22 smithi162 ceph-mon[33787]: pgmap v48: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:26:23.109 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:26:22 smithi142 ceph-mon[29958]: pgmap v48: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:26:23.110 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:26:22 smithi142 ceph-mon[37372]: pgmap v48: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:26:24.709 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:26:24 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: ::ffff:172.21.15.162 - - [06/Sep/2024:14:26:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-06T14:26:24.885 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph health --format=json 2024-09-06T14:26:25.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:26:24 smithi162 ceph-mon[33787]: pgmap v49: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:26:25.148 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:26:25.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:26:24 smithi142 ceph-mon[29958]: pgmap v49: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:26:25.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:26:24 smithi142 ceph-mon[37372]: pgmap v49: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:26:25.659 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:25.649+0000 7f7cd6c2d640 1 -- 172.21.15.142:0/1957628419 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7cd01074e0 msgr2=0x7f7cd010dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:26:25.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:25.649+0000 7f7cd6c2d640 1 --2- 172.21.15.142:0/1957628419 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7cd01074e0 0x7f7cd010dce0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f7cb8009860 tx=0x7f7cb802a340 comp rx=0 tx=0).stop 2024-09-06T14:26:25.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:25.649+0000 7f7cd6c2d640 1 -- 172.21.15.142:0/1957628419 shutdown_connections 2024-09-06T14:26:25.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:25.649+0000 7f7cd6c2d640 1 --2- 172.21.15.142:0/1957628419 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7cd01074e0 0x7f7cd010dce0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:25.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:25.649+0000 7f7cd6c2d640 1 --2- 172.21.15.142:0/1957628419 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7cd0106b60 0x7f7cd0106fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:25.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:25.649+0000 7f7cd6c2d640 1 --2- 172.21.15.142:0/1957628419 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7cd0101530 0x7f7cd0101910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:25.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:25.649+0000 7f7cd6c2d640 1 -- 172.21.15.142:0/1957628419 >> 172.21.15.142:0/1957628419 conn(0x7f7cd0076cf0 msgr2=0x7f7cd0077100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:26:25.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:25.649+0000 7f7cd6c2d640 1 -- 172.21.15.142:0/1957628419 shutdown_connections 2024-09-06T14:26:25.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:25.649+0000 7f7cd6c2d640 1 -- 172.21.15.142:0/1957628419 wait complete. 2024-09-06T14:26:25.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:25.650+0000 7f7cd6c2d640 1 Processor -- start 2024-09-06T14:26:25.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:25.650+0000 7f7cd6c2d640 1 -- start start 2024-09-06T14:26:25.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:25.650+0000 7f7cd6c2d640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7cd0101530 0x7f7cd01964d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:26:25.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:25.650+0000 7f7cd6c2d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7cd0106b60 0x7f7cd0196a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:26:25.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:25.650+0000 7f7cd6c2d640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7cd01074e0 0x7f7cd019bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:26:25.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:25.650+0000 7f7cd6c2d640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f7cd0110040 con 0x7f7cd01074e0 2024-09-06T14:26:25.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:25.650+0000 7f7cd6c2d640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7cd010fec0 con 0x7f7cd0106b60 2024-09-06T14:26:25.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:25.650+0000 7f7cd6c2d640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7f7cd01101c0 con 0x7f7cd0101530 2024-09-06T14:26:25.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:25.650+0000 7f7cd49c9640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7cd0101530 0x7f7cd01964d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:26:25.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:25.650+0000 7f7cd51ca640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7cd01074e0 0x7f7cd019bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:26:25.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:25.650+0000 7f7cd49c9640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7cd0101530 0x7f7cd01964d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:46444/0 (socket says 172.21.15.142:46444) 2024-09-06T14:26:25.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:25.650+0000 7f7cd49c9640 1 -- 172.21.15.142:0/2072300177 learned_addr learned my addr 172.21.15.142:0/2072300177 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:26:25.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:25.650+0000 7f7ccffff640 1 --2- 172.21.15.142:0/2072300177 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7cd0106b60 0x7f7cd0196a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:26:25.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:25.650+0000 7f7cd51ca640 1 -- 172.21.15.142:0/2072300177 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7cd0101530 msgr2=0x7f7cd01964d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:26:25.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:25.650+0000 7f7cd51ca640 1 --2- 172.21.15.142:0/2072300177 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7cd0101530 0x7f7cd01964d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:25.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:25.650+0000 7f7cd51ca640 1 -- 172.21.15.142:0/2072300177 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7cd0106b60 msgr2=0x7f7cd0196a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:26:25.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:25.650+0000 7f7cd51ca640 1 --2- 172.21.15.142:0/2072300177 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7cd0106b60 0x7f7cd0196a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:25.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:25.650+0000 7f7cd51ca640 1 -- 172.21.15.142:0/2072300177 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cd019c580 con 0x7f7cd01074e0 2024-09-06T14:26:25.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:25.650+0000 7f7cd49c9640 1 --2- 172.21.15.142:0/2072300177 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7cd0101530 0x7f7cd01964d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-06T14:26:25.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:25.650+0000 7f7ccffff640 1 --2- 172.21.15.142:0/2072300177 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7cd0106b60 0x7f7cd0196a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-06T14:26:25.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:25.651+0000 7f7cd51ca640 1 --2- 172.21.15.142:0/2072300177 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7cd01074e0 0x7f7cd019bdb0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f7cb802a860 tx=0x7f7cb8033d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:26:25.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:25.651+0000 7f7ccdffb640 1 -- 172.21.15.142:0/2072300177 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cb803f020 con 0x7f7cd01074e0 2024-09-06T14:26:25.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:25.651+0000 7f7ccdffb640 1 -- 172.21.15.142:0/2072300177 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7f7cb80044c0 con 0x7f7cd01074e0 2024-09-06T14:26:25.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:25.651+0000 7f7cd6c2d640 1 -- 172.21.15.142:0/2072300177 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7cd019c810 con 0x7f7cd01074e0 2024-09-06T14:26:25.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:25.651+0000 7f7ccdffb640 1 -- 172.21.15.142:0/2072300177 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cb8037040 con 0x7f7cd01074e0 2024-09-06T14:26:25.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:25.651+0000 7f7cd6c2d640 1 -- 172.21.15.142:0/2072300177 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7cd01a2f30 con 0x7f7cd01074e0 2024-09-06T14:26:25.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:25.652+0000 7f7cd6c2d640 1 -- 172.21.15.142:0/2072300177 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7cd0101910 con 0x7f7cd01074e0 2024-09-06T14:26:25.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:25.653+0000 7f7ccdffb640 1 -- 172.21.15.142:0/2072300177 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 22) ==== 99844+0+0 (secure 0 0 0) 0x7f7cb8042020 con 0x7f7cd01074e0 2024-09-06T14:26:25.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:25.654+0000 7f7ccdffb640 1 --2- 172.21.15.142:0/2072300177 >> v2:172.21.15.142:6800/2944545612 conn(0x7f7cbc077620 0x7f7cbc079ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:26:25.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:25.654+0000 7f7cd49c9640 1 --2- 172.21.15.142:0/2072300177 >> v2:172.21.15.142:6800/2944545612 conn(0x7f7cbc077620 0x7f7cbc079ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:26:25.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:25.654+0000 7f7ccdffb640 1 -- 172.21.15.142:0/2072300177 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 6130+0+0 (secure 0 0 0) 0x7f7cb80b98c0 con 0x7f7cd01074e0 2024-09-06T14:26:25.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:25.654+0000 7f7cd49c9640 1 --2- 172.21.15.142:0/2072300177 >> v2:172.21.15.142:6800/2944545612 conn(0x7f7cbc077620 0x7f7cbc079ae0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7cc800a380 tx=0x7f7cc8011040 comp rx=0 tx=0).ready entity=mgr.14601 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:26:25.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:25.658+0000 7f7ccdffb640 1 -- 172.21.15.142:0/2072300177 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7f7cb8049020 con 0x7f7cd01074e0 2024-09-06T14:26:25.843 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:25.846+0000 7f7cd6c2d640 1 -- 172.21.15.142:0/2072300177 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f7cd01a3100 con 0x7f7cd01074e0 2024-09-06T14:26:25.843 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:25.846+0000 7f7ccdffb640 1 -- 172.21.15.142:0/2072300177 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+212 (secure 0 0 0) 0x7f7cb8082980 con 0x7f7cd01074e0 2024-09-06T14:26:25.844 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:26:25.844 INFO:teuthology.orchestra.run.smithi142.stdout:{"status":"HEALTH_WARN","checks":{"MGR_MODULE_DEPENDENCY":{"severity":"HEALTH_WARN","summary":{"message":"Module 'volumes' has failed dependency: No module named 'ceph.fs'","count":1},"muted":false}},"mutes":[]} 2024-09-06T14:26:25.846 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:25.848+0000 7f7cd6c2d640 1 -- 172.21.15.142:0/2072300177 >> v2:172.21.15.142:6800/2944545612 conn(0x7f7cbc077620 msgr2=0x7f7cbc079ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:26:25.846 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:25.848+0000 7f7cd6c2d640 1 --2- 172.21.15.142:0/2072300177 >> v2:172.21.15.142:6800/2944545612 conn(0x7f7cbc077620 0x7f7cbc079ae0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7cc800a380 tx=0x7f7cc8011040 comp rx=0 tx=0).stop 2024-09-06T14:26:25.846 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:25.848+0000 7f7cd6c2d640 1 -- 172.21.15.142:0/2072300177 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7cd01074e0 msgr2=0x7f7cd019bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:26:25.846 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:25.848+0000 7f7cd6c2d640 1 --2- 172.21.15.142:0/2072300177 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7cd01074e0 0x7f7cd019bdb0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f7cb802a860 tx=0x7f7cb8033d30 comp rx=0 tx=0).stop 2024-09-06T14:26:25.846 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:25.848+0000 7f7cd6c2d640 1 -- 172.21.15.142:0/2072300177 shutdown_connections 2024-09-06T14:26:25.846 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:25.848+0000 7f7cd6c2d640 1 --2- 172.21.15.142:0/2072300177 >> v2:172.21.15.142:6800/2944545612 conn(0x7f7cbc077620 0x7f7cbc079ae0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:25.846 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:25.848+0000 7f7cd6c2d640 1 --2- 172.21.15.142:0/2072300177 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7cd01074e0 0x7f7cd019bdb0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:25.846 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:25.848+0000 7f7cd6c2d640 1 --2- 172.21.15.142:0/2072300177 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7cd0106b60 0x7f7cd0196a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:25.847 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:25.848+0000 7f7cd6c2d640 1 --2- 172.21.15.142:0/2072300177 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7cd0101530 0x7f7cd01964d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:25.847 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:25.848+0000 7f7cd6c2d640 1 -- 172.21.15.142:0/2072300177 >> 172.21.15.142:0/2072300177 conn(0x7f7cd0076cf0 msgr2=0x7f7cd01029f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:26:25.847 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:25.849+0000 7f7cd6c2d640 1 -- 172.21.15.142:0/2072300177 shutdown_connections 2024-09-06T14:26:25.847 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:25.849+0000 7f7cd6c2d640 1 -- 172.21.15.142:0/2072300177 wait complete. 2024-09-06T14:26:26.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:26:26 smithi142 ceph-mon[29958]: pgmap v50: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:26:26.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:26:26 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/2072300177' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:26:26.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:26:26 smithi142 ceph-mon[29958]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-06T14:26:26.959 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:26:26 smithi142 ceph-mon[37372]: pgmap v50: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:26:26.959 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:26:26 smithi142 ceph-mon[37372]: from='client.? 172.21.15.142:0/2072300177' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:26:26.959 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:26:26 smithi142 ceph-mon[37372]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-06T14:26:27.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:26:26 smithi162 ceph-mon[33787]: pgmap v50: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:26:27.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:26:26 smithi162 ceph-mon[33787]: from='client.? 172.21.15.142:0/2072300177' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:26:27.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:26:26 smithi162 ceph-mon[33787]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-06T14:26:27.547 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 06 14:26:27 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-iscsi-iscsi-a[59853]: debug there is no tcmu-runner data available 2024-09-06T14:26:29.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:26:28 smithi162 ceph-mon[33787]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-06T14:26:29.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:26:28 smithi162 ceph-mon[33787]: pgmap v51: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:26:29.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:26:28 smithi142 ceph-mon[37372]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-06T14:26:29.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:26:28 smithi142 ceph-mon[37372]: pgmap v51: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:26:29.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:26:28 smithi142 ceph-mon[29958]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-06T14:26:29.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:26:28 smithi142 ceph-mon[29958]: pgmap v51: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:26:29.716 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph health --format=json 2024-09-06T14:26:29.976 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:26:30.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:30.464+0000 7fc04c6a2640 1 -- 172.21.15.142:0/3130749851 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc044101550 msgr2=0x7fc044101930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:26:30.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:30.464+0000 7fc04c6a2640 1 --2- 172.21.15.142:0/3130749851 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc044101550 0x7fc044101930 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fc040009900 tx=0x7fc04002a3f0 comp rx=0 tx=0).stop 2024-09-06T14:26:30.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:30.464+0000 7fc04c6a2640 1 -- 172.21.15.142:0/3130749851 shutdown_connections 2024-09-06T14:26:30.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:30.464+0000 7fc04c6a2640 1 --2- 172.21.15.142:0/3130749851 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc04410cf90 0x7fc04410f380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:30.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:30.464+0000 7fc04c6a2640 1 --2- 172.21.15.142:0/3130749851 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc04410a660 0x7fc04410ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:30.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:30.464+0000 7fc04c6a2640 1 --2- 172.21.15.142:0/3130749851 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc044101550 0x7fc044101930 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:30.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:30.464+0000 7fc04c6a2640 1 -- 172.21.15.142:0/3130749851 >> 172.21.15.142:0/3130749851 conn(0x7fc044076cf0 msgr2=0x7fc044077100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:26:30.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:30.464+0000 7fc04c6a2640 1 -- 172.21.15.142:0/3130749851 shutdown_connections 2024-09-06T14:26:30.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:30.464+0000 7fc04c6a2640 1 -- 172.21.15.142:0/3130749851 wait complete. 2024-09-06T14:26:30.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:30.465+0000 7fc04c6a2640 1 Processor -- start 2024-09-06T14:26:30.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:30.465+0000 7fc04c6a2640 1 -- start start 2024-09-06T14:26:30.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:30.465+0000 7fc04c6a2640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc044101550 0x7fc0441988a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:26:30.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:30.465+0000 7fc04c6a2640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc04410a660 0x7fc044198de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:26:30.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:30.465+0000 7fc04c6a2640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc04410cf90 0x7fc04419e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:26:30.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:30.465+0000 7fc04c6a2640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fc0441121e0 con 0x7fc04410a660 2024-09-06T14:26:30.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:30.465+0000 7fc04c6a2640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc044112060 con 0x7fc04410cf90 2024-09-06T14:26:30.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:30.465+0000 7fc04c6a2640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7fc044112360 con 0x7fc044101550 2024-09-06T14:26:30.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:30.465+0000 7fc04a43e640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc044101550 0x7fc0441988a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:26:30.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:30.465+0000 7fc049c3d640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc04410a660 0x7fc044198de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:26:30.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:30.465+0000 7fc04a43e640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc044101550 0x7fc0441988a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:47942/0 (socket says 172.21.15.142:47942) 2024-09-06T14:26:30.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:30.465+0000 7fc04a43e640 1 -- 172.21.15.142:0/2836512481 learned_addr learned my addr 172.21.15.142:0/2836512481 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:26:30.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:30.466+0000 7fc04ac3f640 1 --2- 172.21.15.142:0/2836512481 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc04410cf90 0x7fc04419e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:26:30.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:30.466+0000 7fc049c3d640 1 -- 172.21.15.142:0/2836512481 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc044101550 msgr2=0x7fc0441988a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:26:30.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:30.466+0000 7fc049c3d640 1 --2- 172.21.15.142:0/2836512481 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc044101550 0x7fc0441988a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:30.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:30.466+0000 7fc049c3d640 1 -- 172.21.15.142:0/2836512481 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc04410cf90 msgr2=0x7fc04419e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:26:30.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:30.466+0000 7fc049c3d640 1 --2- 172.21.15.142:0/2836512481 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc04410cf90 0x7fc04419e180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:30.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:30.466+0000 7fc049c3d640 1 -- 172.21.15.142:0/2836512481 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc04419e950 con 0x7fc04410a660 2024-09-06T14:26:30.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:30.466+0000 7fc04a43e640 1 --2- 172.21.15.142:0/2836512481 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc044101550 0x7fc0441988a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-06T14:26:30.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:30.466+0000 7fc049c3d640 1 --2- 172.21.15.142:0/2836512481 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc04410a660 0x7fc044198de0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fc03400c970 tx=0x7fc03400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:26:30.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:30.466+0000 7fc03b7fe640 1 -- 172.21.15.142:0/2836512481 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc034007bb0 con 0x7fc04410a660 2024-09-06T14:26:30.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:30.466+0000 7fc04c6a2640 1 -- 172.21.15.142:0/2836512481 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc04419ec40 con 0x7fc04410a660 2024-09-06T14:26:30.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:30.466+0000 7fc03b7fe640 1 -- 172.21.15.142:0/2836512481 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7fc034007d50 con 0x7fc04410a660 2024-09-06T14:26:30.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:30.466+0000 7fc03b7fe640 1 -- 172.21.15.142:0/2836512481 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc034004ca0 con 0x7fc04410a660 2024-09-06T14:26:30.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:30.466+0000 7fc04c6a2640 1 -- 172.21.15.142:0/2836512481 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0441a54d0 con 0x7fc04410a660 2024-09-06T14:26:30.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:30.467+0000 7fc04c6a2640 1 -- 172.21.15.142:0/2836512481 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc04406ab60 con 0x7fc04410a660 2024-09-06T14:26:30.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:30.468+0000 7fc03b7fe640 1 -- 172.21.15.142:0/2836512481 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 22) ==== 99844+0+0 (secure 0 0 0) 0x7fc034004ea0 con 0x7fc04410a660 2024-09-06T14:26:30.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:30.468+0000 7fc03b7fe640 1 --2- 172.21.15.142:0/2836512481 >> v2:172.21.15.142:6800/2944545612 conn(0x7fc030077620 0x7fc030079ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:26:30.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:30.468+0000 7fc04a43e640 1 --2- 172.21.15.142:0/2836512481 >> v2:172.21.15.142:6800/2944545612 conn(0x7fc030077620 0x7fc030079ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:26:30.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:30.468+0000 7fc03b7fe640 1 -- 172.21.15.142:0/2836512481 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 6130+0+0 (secure 0 0 0) 0x7fc0340995d0 con 0x7fc04410a660 2024-09-06T14:26:30.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:30.469+0000 7fc04a43e640 1 --2- 172.21.15.142:0/2836512481 >> v2:172.21.15.142:6800/2944545612 conn(0x7fc030077620 0x7fc030079ae0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc040031440 tx=0x7fc0400058d0 comp rx=0 tx=0).ready entity=mgr.14601 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:26:30.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:30.472+0000 7fc03b7fe640 1 -- 172.21.15.142:0/2836512481 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7fc034014030 con 0x7fc04410a660 2024-09-06T14:26:30.658 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:30.661+0000 7fc04c6a2640 1 -- 172.21.15.142:0/2836512481 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fc0441999d0 con 0x7fc04410a660 2024-09-06T14:26:30.658 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:30.661+0000 7fc03b7fe640 1 -- 172.21.15.142:0/2836512481 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+212 (secure 0 0 0) 0x7fc034066800 con 0x7fc04410a660 2024-09-06T14:26:30.659 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:26:30.659 INFO:teuthology.orchestra.run.smithi142.stdout:{"status":"HEALTH_WARN","checks":{"MGR_MODULE_DEPENDENCY":{"severity":"HEALTH_WARN","summary":{"message":"Module 'volumes' has failed dependency: No module named 'ceph.fs'","count":1},"muted":false}},"mutes":[]} 2024-09-06T14:26:30.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:30.664+0000 7fc04c6a2640 1 -- 172.21.15.142:0/2836512481 >> v2:172.21.15.142:6800/2944545612 conn(0x7fc030077620 msgr2=0x7fc030079ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:26:30.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:30.664+0000 7fc04c6a2640 1 --2- 172.21.15.142:0/2836512481 >> v2:172.21.15.142:6800/2944545612 conn(0x7fc030077620 0x7fc030079ae0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc040031440 tx=0x7fc0400058d0 comp rx=0 tx=0).stop 2024-09-06T14:26:30.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:30.664+0000 7fc04c6a2640 1 -- 172.21.15.142:0/2836512481 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc04410a660 msgr2=0x7fc044198de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:26:30.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:30.664+0000 7fc04c6a2640 1 --2- 172.21.15.142:0/2836512481 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc04410a660 0x7fc044198de0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fc03400c970 tx=0x7fc03400ce30 comp rx=0 tx=0).stop 2024-09-06T14:26:30.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:30.664+0000 7fc04c6a2640 1 -- 172.21.15.142:0/2836512481 shutdown_connections 2024-09-06T14:26:30.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:30.664+0000 7fc04c6a2640 1 --2- 172.21.15.142:0/2836512481 >> v2:172.21.15.142:6800/2944545612 conn(0x7fc030077620 0x7fc030079ae0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:30.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:30.664+0000 7fc04c6a2640 1 --2- 172.21.15.142:0/2836512481 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc04410cf90 0x7fc04419e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:30.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:30.664+0000 7fc04c6a2640 1 --2- 172.21.15.142:0/2836512481 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc04410a660 0x7fc044198de0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:30.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:30.664+0000 7fc04c6a2640 1 --2- 172.21.15.142:0/2836512481 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc044101550 0x7fc0441988a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:30.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:30.664+0000 7fc04c6a2640 1 -- 172.21.15.142:0/2836512481 >> 172.21.15.142:0/2836512481 conn(0x7fc044076cf0 msgr2=0x7fc0441002e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:26:30.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:30.664+0000 7fc04c6a2640 1 -- 172.21.15.142:0/2836512481 shutdown_connections 2024-09-06T14:26:30.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:30.664+0000 7fc04c6a2640 1 -- 172.21.15.142:0/2836512481 wait complete. 2024-09-06T14:26:30.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:26:30 smithi142 ceph-mon[29958]: pgmap v52: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:26:30.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:26:30 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/2836512481' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:26:30.959 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:26:30 smithi142 ceph-mon[37372]: pgmap v52: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:26:30.959 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:26:30 smithi142 ceph-mon[37372]: from='client.? 172.21.15.142:0/2836512481' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:26:31.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:26:30 smithi162 ceph-mon[33787]: pgmap v52: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:26:31.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:26:30 smithi162 ceph-mon[33787]: from='client.? 172.21.15.142:0/2836512481' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:26:33.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:26:32 smithi162 ceph-mon[33787]: pgmap v53: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:26:33.165 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:26:32 smithi142 ceph-mon[29958]: pgmap v53: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:26:33.165 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:26:32 smithi142 ceph-mon[37372]: pgmap v53: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:26:34.389 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph health --format=json 2024-09-06T14:26:34.651 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:26:34.709 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:26:34 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: ::ffff:172.21.15.162 - - [06/Sep/2024:14:26:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-06T14:26:35.033 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:26:34 smithi142 ceph-mon[29958]: pgmap v54: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:26:35.033 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:26:34 smithi142 ceph-mon[37372]: pgmap v54: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:26:35.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:26:34 smithi162 ceph-mon[33787]: pgmap v54: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:26:35.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:35.114+0000 7f2aba462640 1 -- 172.21.15.142:0/2888613567 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2ab4112ef0 msgr2=0x7f2ab4113330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:26:35.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:35.114+0000 7f2aba462640 1 --2- 172.21.15.142:0/2888613567 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2ab4112ef0 0x7f2ab4113330 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2a9c009860 tx=0x7f2a9c02a340 comp rx=0 tx=0).stop 2024-09-06T14:26:35.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:35.114+0000 7f2aba462640 1 -- 172.21.15.142:0/2888613567 shutdown_connections 2024-09-06T14:26:35.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:35.114+0000 7f2aba462640 1 --2- 172.21.15.142:0/2888613567 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2ab4113870 0x7f2ab411a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:35.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:35.114+0000 7f2aba462640 1 --2- 172.21.15.142:0/2888613567 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2ab4112ef0 0x7f2ab4113330 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:35.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:35.114+0000 7f2aba462640 1 --2- 172.21.15.142:0/2888613567 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2ab410d4e0 0x7f2ab410d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:35.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:35.114+0000 7f2aba462640 1 -- 172.21.15.142:0/2888613567 >> 172.21.15.142:0/2888613567 conn(0x7f2ab406d9e0 msgr2=0x7f2ab406ddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:26:35.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:35.114+0000 7f2aba462640 1 -- 172.21.15.142:0/2888613567 shutdown_connections 2024-09-06T14:26:35.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:35.114+0000 7f2aba462640 1 -- 172.21.15.142:0/2888613567 wait complete. 2024-09-06T14:26:35.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:35.114+0000 7f2aba462640 1 Processor -- start 2024-09-06T14:26:35.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:35.114+0000 7f2aba462640 1 -- start start 2024-09-06T14:26:35.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:35.115+0000 7f2aba462640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2ab410d4e0 0x7f2ab41a30f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:26:35.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:35.115+0000 7f2aba462640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2ab4112ef0 0x7f2ab41a3630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:26:35.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:35.115+0000 7f2aba462640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2ab4113870 0x7f2ab41a89d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:26:35.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:35.115+0000 7f2aba462640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f2ab411cb70 con 0x7f2ab4113870 2024-09-06T14:26:35.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:35.115+0000 7f2aba462640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f2ab411c9f0 con 0x7f2ab410d4e0 2024-09-06T14:26:35.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:35.115+0000 7f2aba462640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7f2ab411ccf0 con 0x7f2ab4112ef0 2024-09-06T14:26:35.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:35.115+0000 7f2aabfff640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2ab4112ef0 0x7f2ab41a3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:26:35.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:35.115+0000 7f2ab89ff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2ab4113870 0x7f2ab41a89d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:26:35.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:35.115+0000 7f2ab89ff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2ab4113870 0x7f2ab41a89d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:36130/0 (socket says 172.21.15.142:36130) 2024-09-06T14:26:35.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:35.115+0000 7f2aabfff640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2ab4112ef0 0x7f2ab41a3630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:47958/0 (socket says 172.21.15.142:47958) 2024-09-06T14:26:35.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:35.115+0000 7f2ab89ff640 1 -- 172.21.15.142:0/3604166178 learned_addr learned my addr 172.21.15.142:0/3604166178 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:26:35.135 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:35.115+0000 7f2ab3fff640 1 --2- 172.21.15.142:0/3604166178 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2ab410d4e0 0x7f2ab41a30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:26:35.135 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:35.115+0000 7f2ab89ff640 1 -- 172.21.15.142:0/3604166178 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2ab4112ef0 msgr2=0x7f2ab41a3630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:26:35.135 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:35.115+0000 7f2ab89ff640 1 --2- 172.21.15.142:0/3604166178 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2ab4112ef0 0x7f2ab41a3630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:35.135 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:35.115+0000 7f2ab89ff640 1 -- 172.21.15.142:0/3604166178 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2ab410d4e0 msgr2=0x7f2ab41a30f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:26:35.135 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:35.115+0000 7f2ab89ff640 1 --2- 172.21.15.142:0/3604166178 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2ab410d4e0 0x7f2ab41a30f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:35.135 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:35.115+0000 7f2ab89ff640 1 -- 172.21.15.142:0/3604166178 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ab41a91a0 con 0x7f2ab4113870 2024-09-06T14:26:35.135 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:35.115+0000 7f2aabfff640 1 --2- 172.21.15.142:0/3604166178 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2ab4112ef0 0x7f2ab41a3630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-06T14:26:35.135 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:35.115+0000 7f2ab89ff640 1 --2- 172.21.15.142:0/3604166178 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2ab4113870 0x7f2ab41a89d0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f2aac00cad0 tx=0x7f2aac00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:26:35.135 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:35.115+0000 7f2ab1ffb640 1 -- 172.21.15.142:0/3604166178 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aac00ae20 con 0x7f2ab4113870 2024-09-06T14:26:35.135 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:35.115+0000 7f2ab1ffb640 1 -- 172.21.15.142:0/3604166178 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7f2aac005020 con 0x7f2ab4113870 2024-09-06T14:26:35.135 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:35.115+0000 7f2aba462640 1 -- 172.21.15.142:0/3604166178 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ab41a9490 con 0x7f2ab4113870 2024-09-06T14:26:35.135 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:35.115+0000 7f2ab1ffb640 1 -- 172.21.15.142:0/3604166178 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aac00f6b0 con 0x7f2ab4113870 2024-09-06T14:26:35.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:35.116+0000 7f2aba462640 1 -- 172.21.15.142:0/3604166178 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ab41afb50 con 0x7f2ab4113870 2024-09-06T14:26:35.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:35.117+0000 7f2aba462640 1 -- 172.21.15.142:0/3604166178 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ab410d8c0 con 0x7f2ab4113870 2024-09-06T14:26:35.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:35.117+0000 7f2ab1ffb640 1 -- 172.21.15.142:0/3604166178 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 22) ==== 99844+0+0 (secure 0 0 0) 0x7f2aac004a40 con 0x7f2ab4113870 2024-09-06T14:26:35.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:35.118+0000 7f2ab1ffb640 1 --2- 172.21.15.142:0/3604166178 >> v2:172.21.15.142:6800/2944545612 conn(0x7f2aa0077620 0x7f2aa0079ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:26:35.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:35.118+0000 7f2ab3fff640 1 --2- 172.21.15.142:0/3604166178 >> v2:172.21.15.142:6800/2944545612 conn(0x7f2aa0077620 0x7f2aa0079ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:26:35.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:35.118+0000 7f2ab1ffb640 1 -- 172.21.15.142:0/3604166178 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 6130+0+0 (secure 0 0 0) 0x7f2aac0998c0 con 0x7f2ab4113870 2024-09-06T14:26:35.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:35.118+0000 7f2ab3fff640 1 --2- 172.21.15.142:0/3604166178 >> v2:172.21.15.142:6800/2944545612 conn(0x7f2aa0077620 0x7f2aa0079ae0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2aa4005fd0 tx=0x7f2aa4009260 comp rx=0 tx=0).ready entity=mgr.14601 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:26:35.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:35.122+0000 7f2ab1ffb640 1 -- 172.21.15.142:0/3604166178 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7f2aac014030 con 0x7f2ab4113870 2024-09-06T14:26:35.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:35.310+0000 7f2aba462640 1 -- 172.21.15.142:0/3604166178 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f2ab41afde0 con 0x7f2ab4113870 2024-09-06T14:26:35.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:35.310+0000 7f2ab1ffb640 1 -- 172.21.15.142:0/3604166178 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+212 (secure 0 0 0) 0x7f2aac066b70 con 0x7f2ab4113870 2024-09-06T14:26:35.308 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:26:35.308 INFO:teuthology.orchestra.run.smithi142.stdout:{"status":"HEALTH_WARN","checks":{"MGR_MODULE_DEPENDENCY":{"severity":"HEALTH_WARN","summary":{"message":"Module 'volumes' has failed dependency: No module named 'ceph.fs'","count":1},"muted":false}},"mutes":[]} 2024-09-06T14:26:35.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:35.312+0000 7f2aba462640 1 -- 172.21.15.142:0/3604166178 >> v2:172.21.15.142:6800/2944545612 conn(0x7f2aa0077620 msgr2=0x7f2aa0079ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:26:35.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:35.312+0000 7f2aba462640 1 --2- 172.21.15.142:0/3604166178 >> v2:172.21.15.142:6800/2944545612 conn(0x7f2aa0077620 0x7f2aa0079ae0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2aa4005fd0 tx=0x7f2aa4009260 comp rx=0 tx=0).stop 2024-09-06T14:26:35.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:35.312+0000 7f2aba462640 1 -- 172.21.15.142:0/3604166178 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2ab4113870 msgr2=0x7f2ab41a89d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:26:35.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:35.312+0000 7f2aba462640 1 --2- 172.21.15.142:0/3604166178 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2ab4113870 0x7f2ab41a89d0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f2aac00cad0 tx=0x7f2aac00cf90 comp rx=0 tx=0).stop 2024-09-06T14:26:35.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:35.312+0000 7f2aba462640 1 -- 172.21.15.142:0/3604166178 shutdown_connections 2024-09-06T14:26:35.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:35.312+0000 7f2aba462640 1 --2- 172.21.15.142:0/3604166178 >> v2:172.21.15.142:6800/2944545612 conn(0x7f2aa0077620 0x7f2aa0079ae0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:35.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:35.312+0000 7f2aba462640 1 --2- 172.21.15.142:0/3604166178 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2ab4113870 0x7f2ab41a89d0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:35.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:35.312+0000 7f2aba462640 1 --2- 172.21.15.142:0/3604166178 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2ab4112ef0 0x7f2ab41a3630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:35.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:35.312+0000 7f2aba462640 1 --2- 172.21.15.142:0/3604166178 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2ab410d4e0 0x7f2ab41a30f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:35.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:35.312+0000 7f2aba462640 1 -- 172.21.15.142:0/3604166178 >> 172.21.15.142:0/3604166178 conn(0x7f2ab406d9e0 msgr2=0x7f2ab410bec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:26:35.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:35.313+0000 7f2aba462640 1 -- 172.21.15.142:0/3604166178 shutdown_connections 2024-09-06T14:26:35.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:35.313+0000 7f2aba462640 1 -- 172.21.15.142:0/3604166178 wait complete. 2024-09-06T14:26:36.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:26:35 smithi162 ceph-mon[33787]: from='client.? 172.21.15.142:0/3604166178' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:26:36.064 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:26:35 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/3604166178' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:26:36.064 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:26:35 smithi142 ceph-mon[37372]: from='client.? 172.21.15.142:0/3604166178' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:26:37.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:26:36 smithi162 ceph-mon[33787]: pgmap v55: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:26:37.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:26:36 smithi142 ceph-mon[29958]: pgmap v55: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:26:37.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:26:36 smithi142 ceph-mon[37372]: pgmap v55: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:26:37.547 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 06 14:26:37 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-iscsi-iscsi-a[59853]: debug there is no tcmu-runner data available 2024-09-06T14:26:39.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:26:38 smithi162 ceph-mon[33787]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-06T14:26:39.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:26:38 smithi162 ceph-mon[33787]: pgmap v56: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:26:39.099 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph health --format=json 2024-09-06T14:26:39.128 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:26:38 smithi142 ceph-mon[29958]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-06T14:26:39.128 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:26:38 smithi142 ceph-mon[29958]: pgmap v56: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:26:39.128 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:26:38 smithi142 ceph-mon[37372]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-06T14:26:39.129 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:26:38 smithi142 ceph-mon[37372]: pgmap v56: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:26:39.361 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:26:39.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:39.881+0000 7faddbd09640 1 -- 172.21.15.142:0/2709386422 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fadd4108e70 msgr2=0x7fadd410f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:26:39.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:39.881+0000 7faddbd09640 1 --2- 172.21.15.142:0/2709386422 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fadd4108e70 0x7fadd410f670 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fadcc009ca0 tx=0x7fadcc02a8c0 comp rx=0 tx=0).stop 2024-09-06T14:26:39.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:39.881+0000 7faddbd09640 1 -- 172.21.15.142:0/2709386422 shutdown_connections 2024-09-06T14:26:39.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:39.881+0000 7faddbd09640 1 --2- 172.21.15.142:0/2709386422 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fadd4108e70 0x7fadd410f670 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:39.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:39.881+0000 7faddbd09640 1 --2- 172.21.15.142:0/2709386422 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fadd41084f0 0x7fadd4108930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:39.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:39.881+0000 7faddbd09640 1 --2- 172.21.15.142:0/2709386422 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fadd4102ae0 0x7fadd4102ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:39.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:39.881+0000 7faddbd09640 1 -- 172.21.15.142:0/2709386422 >> 172.21.15.142:0/2709386422 conn(0x7fadd40fe530 msgr2=0x7fadd4100950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:26:39.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:39.881+0000 7faddbd09640 1 -- 172.21.15.142:0/2709386422 shutdown_connections 2024-09-06T14:26:39.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:39.881+0000 7faddbd09640 1 -- 172.21.15.142:0/2709386422 wait complete. 2024-09-06T14:26:39.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:39.882+0000 7faddbd09640 1 Processor -- start 2024-09-06T14:26:39.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:39.882+0000 7faddbd09640 1 -- start start 2024-09-06T14:26:39.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:39.882+0000 7faddbd09640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fadd4102ae0 0x7fadd4075200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:26:39.902 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:39.882+0000 7faddbd09640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fadd41084f0 0x7fadd4075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:26:39.902 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:39.882+0000 7faddbd09640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fadd4108e70 0x7fadd4075c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:26:39.902 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:39.882+0000 7faddbd09640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fadd4112240 con 0x7fadd4108e70 2024-09-06T14:26:39.902 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:39.882+0000 7faddbd09640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fadd41120c0 con 0x7fadd4102ae0 2024-09-06T14:26:39.902 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:39.882+0000 7faddbd09640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7fadd41123c0 con 0x7fadd41084f0 2024-09-06T14:26:39.902 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:39.882+0000 7fadda2a6640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fadd4108e70 0x7fadd4075c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:26:39.902 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:39.882+0000 7fadd92a4640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fadd41084f0 0x7fadd4075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:26:39.902 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:39.882+0000 7fadda2a6640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fadd4108e70 0x7fadd4075c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:36600/0 (socket says 172.21.15.142:36600) 2024-09-06T14:26:39.902 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:39.882+0000 7fadda2a6640 1 -- 172.21.15.142:0/580984720 learned_addr learned my addr 172.21.15.142:0/580984720 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:26:39.902 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:39.882+0000 7fadd9aa5640 1 --2- 172.21.15.142:0/580984720 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fadd4102ae0 0x7fadd4075200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:26:39.902 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:39.882+0000 7fadd92a4640 1 -- 172.21.15.142:0/580984720 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fadd4102ae0 msgr2=0x7fadd4075200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:26:39.902 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:39.882+0000 7fadd92a4640 1 --2- 172.21.15.142:0/580984720 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fadd4102ae0 0x7fadd4075200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:39.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:39.882+0000 7fadd92a4640 1 -- 172.21.15.142:0/580984720 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fadd4108e70 msgr2=0x7fadd4075c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:26:39.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:39.882+0000 7fadd92a4640 1 --2- 172.21.15.142:0/580984720 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fadd4108e70 0x7fadd4075c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:39.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:39.882+0000 7fadd92a4640 1 -- 172.21.15.142:0/580984720 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadd41a69f0 con 0x7fadd41084f0 2024-09-06T14:26:39.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:39.882+0000 7fadda2a6640 1 --2- 172.21.15.142:0/580984720 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fadd4108e70 0x7fadd4075c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-06T14:26:39.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:39.882+0000 7fadd9aa5640 1 --2- 172.21.15.142:0/580984720 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fadd4102ae0 0x7fadd4075200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-06T14:26:39.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:39.883+0000 7fadd92a4640 1 --2- 172.21.15.142:0/580984720 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fadd41084f0 0x7fadd4075740 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fadd000c970 tx=0x7fadd000ce30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:26:39.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:39.883+0000 7fadcaffd640 1 -- 172.21.15.142:0/580984720 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadd0007bb0 con 0x7fadd41084f0 2024-09-06T14:26:39.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:39.883+0000 7fadcaffd640 1 -- 172.21.15.142:0/580984720 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7fadd0007d50 con 0x7fadd41084f0 2024-09-06T14:26:39.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:39.883+0000 7faddbd09640 1 -- 172.21.15.142:0/580984720 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fadd41a6ce0 con 0x7fadd41084f0 2024-09-06T14:26:39.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:39.883+0000 7fadcaffd640 1 -- 172.21.15.142:0/580984720 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadd0018660 con 0x7fadd41084f0 2024-09-06T14:26:39.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:39.883+0000 7faddbd09640 1 -- 172.21.15.142:0/580984720 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fadd41a7210 con 0x7fadd41084f0 2024-09-06T14:26:39.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:39.884+0000 7faddbd09640 1 -- 172.21.15.142:0/580984720 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fadd4102ec0 con 0x7fadd41084f0 2024-09-06T14:26:39.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:39.884+0000 7fadcaffd640 1 -- 172.21.15.142:0/580984720 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 22) ==== 99844+0+0 (secure 0 0 0) 0x7fadd000f450 con 0x7fadd41084f0 2024-09-06T14:26:39.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:39.885+0000 7fadcaffd640 1 --2- 172.21.15.142:0/580984720 >> v2:172.21.15.142:6800/2944545612 conn(0x7fadc0077620 0x7fadc0079ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:26:39.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:39.885+0000 7fadcaffd640 1 -- 172.21.15.142:0/580984720 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(64..64 src has 1..64) ==== 6130+0+0 (secure 0 0 0) 0x7fadd0099970 con 0x7fadd41084f0 2024-09-06T14:26:39.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:39.885+0000 7fadd9aa5640 1 --2- 172.21.15.142:0/580984720 >> v2:172.21.15.142:6800/2944545612 conn(0x7fadc0077620 0x7fadc0079ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:26:39.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:39.885+0000 7fadd9aa5640 1 --2- 172.21.15.142:0/580984720 >> v2:172.21.15.142:6800/2944545612 conn(0x7fadc0077620 0x7fadc0079ae0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fadc4009c30 tx=0x7fadc4005f90 comp rx=0 tx=0).ready entity=mgr.14601 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:26:39.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:39.889+0000 7fadcaffd640 1 -- 172.21.15.142:0/580984720 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7fadd0014030 con 0x7fadd41084f0 2024-09-06T14:26:40.080 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:40.081+0000 7faddbd09640 1 -- 172.21.15.142:0/580984720 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fadd41a74a0 con 0x7fadd41084f0 2024-09-06T14:26:40.080 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:40.082+0000 7fadcaffd640 1 -- 172.21.15.142:0/580984720 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+212 (secure 0 0 0) 0x7fadd0062ab0 con 0x7fadd41084f0 2024-09-06T14:26:40.081 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:26:40.081 INFO:teuthology.orchestra.run.smithi142.stdout:{"status":"HEALTH_WARN","checks":{"MGR_MODULE_DEPENDENCY":{"severity":"HEALTH_WARN","summary":{"message":"Module 'volumes' has failed dependency: No module named 'ceph.fs'","count":1},"muted":false}},"mutes":[]} 2024-09-06T14:26:40.082 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:40.084+0000 7faddbd09640 1 -- 172.21.15.142:0/580984720 >> v2:172.21.15.142:6800/2944545612 conn(0x7fadc0077620 msgr2=0x7fadc0079ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:26:40.082 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:40.084+0000 7faddbd09640 1 --2- 172.21.15.142:0/580984720 >> v2:172.21.15.142:6800/2944545612 conn(0x7fadc0077620 0x7fadc0079ae0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fadc4009c30 tx=0x7fadc4005f90 comp rx=0 tx=0).stop 2024-09-06T14:26:40.082 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:40.084+0000 7faddbd09640 1 -- 172.21.15.142:0/580984720 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fadd41084f0 msgr2=0x7fadd4075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:26:40.082 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:40.084+0000 7faddbd09640 1 --2- 172.21.15.142:0/580984720 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fadd41084f0 0x7fadd4075740 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fadd000c970 tx=0x7fadd000ce30 comp rx=0 tx=0).stop 2024-09-06T14:26:40.083 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:40.084+0000 7faddbd09640 1 -- 172.21.15.142:0/580984720 shutdown_connections 2024-09-06T14:26:40.083 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:40.084+0000 7faddbd09640 1 --2- 172.21.15.142:0/580984720 >> v2:172.21.15.142:6800/2944545612 conn(0x7fadc0077620 0x7fadc0079ae0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:40.083 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:40.084+0000 7faddbd09640 1 --2- 172.21.15.142:0/580984720 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fadd4108e70 0x7fadd4075c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:40.084 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:40.084+0000 7faddbd09640 1 --2- 172.21.15.142:0/580984720 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fadd41084f0 0x7fadd4075740 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:40.084 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:40.084+0000 7faddbd09640 1 --2- 172.21.15.142:0/580984720 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fadd4102ae0 0x7fadd4075200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:40.084 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:40.084+0000 7faddbd09640 1 -- 172.21.15.142:0/580984720 >> 172.21.15.142:0/580984720 conn(0x7fadd40fe530 msgr2=0x7fadd4101a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:26:40.084 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:40.084+0000 7faddbd09640 1 -- 172.21.15.142:0/580984720 shutdown_connections 2024-09-06T14:26:40.084 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:40.084+0000 7faddbd09640 1 -- 172.21.15.142:0/580984720 wait complete. 2024-09-06T14:26:41.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:26:40 smithi162 ceph-mon[33787]: pgmap v57: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:26:41.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:26:40 smithi162 ceph-mon[33787]: from='client.? 172.21.15.142:0/580984720' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:26:41.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:26:40 smithi142 ceph-mon[29958]: pgmap v57: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:26:41.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:26:40 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/580984720' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:26:41.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:26:40 smithi142 ceph-mon[37372]: pgmap v57: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:26:41.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:26:40 smithi142 ceph-mon[37372]: from='client.? 172.21.15.142:0/580984720' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:26:42.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:26:41 smithi162 ceph-mon[33787]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-06T14:26:42.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:26:41 smithi142 ceph-mon[29958]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-06T14:26:42.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:26:41 smithi142 ceph-mon[37372]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-06T14:26:43.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:26:42 smithi162 ceph-mon[33787]: pgmap v58: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:26:43.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:26:42 smithi142 ceph-mon[29958]: pgmap v58: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:26:43.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:26:42 smithi142 ceph-mon[37372]: pgmap v58: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:26:43.846 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph health --format=json 2024-09-06T14:26:44.109 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:26:44.554 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:26:44 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: ::ffff:172.21.15.162 - - [06/Sep/2024:14:26:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-06T14:26:44.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:44.556+0000 7fa2f5e07640 1 -- 172.21.15.142:0/2754898024 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa2f01042f0 msgr2=0x7fa2f010c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:26:44.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:44.556+0000 7fa2f5e07640 1 --2- 172.21.15.142:0/2754898024 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa2f01042f0 0x7fa2f010c7e0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fa2d8009900 tx=0x7fa2d802a410 comp rx=0 tx=0).stop 2024-09-06T14:26:44.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:44.556+0000 7fa2f5e07640 1 -- 172.21.15.142:0/2754898024 shutdown_connections 2024-09-06T14:26:44.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:44.556+0000 7fa2f5e07640 1 --2- 172.21.15.142:0/2754898024 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa2f010ce30 0x7fa2f010f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:44.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:44.556+0000 7fa2f5e07640 1 --2- 172.21.15.142:0/2754898024 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa2f01042f0 0x7fa2f010c7e0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:44.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:44.556+0000 7fa2f5e07640 1 --2- 172.21.15.142:0/2754898024 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa2f01039d0 0x7fa2f0103db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:44.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:44.556+0000 7fa2f5e07640 1 -- 172.21.15.142:0/2754898024 >> 172.21.15.142:0/2754898024 conn(0x7fa2f00fd430 msgr2=0x7fa2f00ff850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:26:44.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:44.556+0000 7fa2f5e07640 1 -- 172.21.15.142:0/2754898024 shutdown_connections 2024-09-06T14:26:44.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:44.556+0000 7fa2f5e07640 1 -- 172.21.15.142:0/2754898024 wait complete. 2024-09-06T14:26:44.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:44.556+0000 7fa2f5e07640 1 Processor -- start 2024-09-06T14:26:44.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:44.557+0000 7fa2f5e07640 1 -- start start 2024-09-06T14:26:44.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:44.557+0000 7fa2f5e07640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa2f01039d0 0x7fa2f010bba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:26:44.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:44.557+0000 7fa2f5e07640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa2f01042f0 0x7fa2f010c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:26:44.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:44.557+0000 7fa2f5e07640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa2f010ce30 0x7fa2f01a1380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:26:44.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:44.557+0000 7fa2f5e07640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fa2f01121b0 con 0x7fa2f01039d0 2024-09-06T14:26:44.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:44.557+0000 7fa2f5e07640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa2f0112030 con 0x7fa2f01042f0 2024-09-06T14:26:44.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:44.557+0000 7fa2f5e07640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7fa2f0112330 con 0x7fa2f010ce30 2024-09-06T14:26:44.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:44.557+0000 7fa2ef7fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa2f01039d0 0x7fa2f010bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:26:44.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:44.557+0000 7fa2ef7fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa2f01039d0 0x7fa2f010bba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:36616/0 (socket says 172.21.15.142:36616) 2024-09-06T14:26:44.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:44.557+0000 7fa2ef7fe640 1 -- 172.21.15.142:0/3136412482 learned_addr learned my addr 172.21.15.142:0/3136412482 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:26:44.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:44.557+0000 7fa2effff640 1 --2- 172.21.15.142:0/3136412482 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa2f010ce30 0x7fa2f01a1380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:26:44.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:44.557+0000 7fa2e7fff640 1 --2- 172.21.15.142:0/3136412482 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa2f01042f0 0x7fa2f010c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:26:44.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:44.557+0000 7fa2ef7fe640 1 -- 172.21.15.142:0/3136412482 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa2f010ce30 msgr2=0x7fa2f01a1380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:26:44.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:44.557+0000 7fa2ef7fe640 1 --2- 172.21.15.142:0/3136412482 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa2f010ce30 0x7fa2f01a1380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:44.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:44.557+0000 7fa2ef7fe640 1 -- 172.21.15.142:0/3136412482 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa2f01042f0 msgr2=0x7fa2f010c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:26:44.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:44.557+0000 7fa2ef7fe640 1 --2- 172.21.15.142:0/3136412482 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa2f01042f0 0x7fa2f010c0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:44.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:44.557+0000 7fa2ef7fe640 1 -- 172.21.15.142:0/3136412482 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2f01a1b00 con 0x7fa2f01039d0 2024-09-06T14:26:44.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:44.557+0000 7fa2e7fff640 1 --2- 172.21.15.142:0/3136412482 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa2f01042f0 0x7fa2f010c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-06T14:26:44.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:44.557+0000 7fa2ef7fe640 1 --2- 172.21.15.142:0/3136412482 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa2f01039d0 0x7fa2f010bba0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fa2e000ea60 tx=0x7fa2e000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:26:44.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:44.558+0000 7fa2ed7fa640 1 -- 172.21.15.142:0/3136412482 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2e000cde0 con 0x7fa2f01039d0 2024-09-06T14:26:44.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:44.558+0000 7fa2f5e07640 1 -- 172.21.15.142:0/3136412482 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2f01a1df0 con 0x7fa2f01039d0 2024-09-06T14:26:44.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:44.558+0000 7fa2ed7fa640 1 -- 172.21.15.142:0/3136412482 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7fa2e00035f0 con 0x7fa2f01039d0 2024-09-06T14:26:44.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:44.558+0000 7fa2ed7fa640 1 -- 172.21.15.142:0/3136412482 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2e00044a0 con 0x7fa2f01039d0 2024-09-06T14:26:44.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:44.558+0000 7fa2f5e07640 1 -- 172.21.15.142:0/3136412482 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2f01ab740 con 0x7fa2f01039d0 2024-09-06T14:26:44.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:44.559+0000 7fa2f5e07640 1 -- 172.21.15.142:0/3136412482 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2f0103db0 con 0x7fa2f01039d0 2024-09-06T14:26:44.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:44.559+0000 7fa2ed7fa640 1 -- 172.21.15.142:0/3136412482 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 22) ==== 99844+0+0 (secure 0 0 0) 0x7fa2e0004000 con 0x7fa2f01039d0 2024-09-06T14:26:44.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:44.560+0000 7fa2ed7fa640 1 --2- 172.21.15.142:0/3136412482 >> v2:172.21.15.142:6800/2944545612 conn(0x7fa2dc077620 0x7fa2dc079ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:26:44.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:44.560+0000 7fa2e7fff640 1 --2- 172.21.15.142:0/3136412482 >> v2:172.21.15.142:6800/2944545612 conn(0x7fa2dc077620 0x7fa2dc079ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:26:44.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:44.560+0000 7fa2ed7fa640 1 -- 172.21.15.142:0/3136412482 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 6130+0+0 (secure 0 0 0) 0x7fa2e0099d80 con 0x7fa2f01039d0 2024-09-06T14:26:44.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:44.560+0000 7fa2e7fff640 1 --2- 172.21.15.142:0/3136412482 >> v2:172.21.15.142:6800/2944545612 conn(0x7fa2dc077620 0x7fa2dc079ae0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa2d8000c00 tx=0x7fa2d8031780 comp rx=0 tx=0).ready entity=mgr.14601 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:26:44.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:44.564+0000 7fa2ed7fa640 1 -- 172.21.15.142:0/3136412482 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7fa2e0062e40 con 0x7fa2f01039d0 2024-09-06T14:26:44.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:44.751+0000 7fa2f5e07640 1 -- 172.21.15.142:0/3136412482 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fa2f01ab9d0 con 0x7fa2f01039d0 2024-09-06T14:26:44.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:44.751+0000 7fa2ed7fa640 1 -- 172.21.15.142:0/3136412482 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+212 (secure 0 0 0) 0x7fa2e0083e20 con 0x7fa2f01039d0 2024-09-06T14:26:44.749 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:26:44.749 INFO:teuthology.orchestra.run.smithi142.stdout:{"status":"HEALTH_WARN","checks":{"MGR_MODULE_DEPENDENCY":{"severity":"HEALTH_WARN","summary":{"message":"Module 'volumes' has failed dependency: No module named 'ceph.fs'","count":1},"muted":false}},"mutes":[]} 2024-09-06T14:26:44.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:44.753+0000 7fa2f5e07640 1 -- 172.21.15.142:0/3136412482 >> v2:172.21.15.142:6800/2944545612 conn(0x7fa2dc077620 msgr2=0x7fa2dc079ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:26:44.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:44.753+0000 7fa2f5e07640 1 --2- 172.21.15.142:0/3136412482 >> v2:172.21.15.142:6800/2944545612 conn(0x7fa2dc077620 0x7fa2dc079ae0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa2d8000c00 tx=0x7fa2d8031780 comp rx=0 tx=0).stop 2024-09-06T14:26:44.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:44.753+0000 7fa2f5e07640 1 -- 172.21.15.142:0/3136412482 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa2f01039d0 msgr2=0x7fa2f010bba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:26:44.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:44.753+0000 7fa2f5e07640 1 --2- 172.21.15.142:0/3136412482 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa2f01039d0 0x7fa2f010bba0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fa2e000ea60 tx=0x7fa2e000ef20 comp rx=0 tx=0).stop 2024-09-06T14:26:44.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:44.753+0000 7fa2f5e07640 1 -- 172.21.15.142:0/3136412482 shutdown_connections 2024-09-06T14:26:44.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:44.753+0000 7fa2f5e07640 1 --2- 172.21.15.142:0/3136412482 >> v2:172.21.15.142:6800/2944545612 conn(0x7fa2dc077620 0x7fa2dc079ae0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:44.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:44.753+0000 7fa2f5e07640 1 --2- 172.21.15.142:0/3136412482 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa2f010ce30 0x7fa2f01a1380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:44.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:44.753+0000 7fa2f5e07640 1 --2- 172.21.15.142:0/3136412482 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa2f01042f0 0x7fa2f010c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:44.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:44.753+0000 7fa2f5e07640 1 --2- 172.21.15.142:0/3136412482 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa2f01039d0 0x7fa2f010bba0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:44.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:44.753+0000 7fa2f5e07640 1 -- 172.21.15.142:0/3136412482 >> 172.21.15.142:0/3136412482 conn(0x7fa2f00fd430 msgr2=0x7fa2f00fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:26:44.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:44.753+0000 7fa2f5e07640 1 -- 172.21.15.142:0/3136412482 shutdown_connections 2024-09-06T14:26:44.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:44.753+0000 7fa2f5e07640 1 -- 172.21.15.142:0/3136412482 wait complete. 2024-09-06T14:26:44.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:26:44 smithi142 ceph-mon[29958]: pgmap v59: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:26:44.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:26:44 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/3136412482' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:26:44.959 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:26:44 smithi142 ceph-mon[37372]: pgmap v59: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:26:44.960 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:26:44 smithi142 ceph-mon[37372]: from='client.? 172.21.15.142:0/3136412482' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:26:45.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:26:44 smithi162 ceph-mon[33787]: pgmap v59: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:26:45.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:26:44 smithi162 ceph-mon[33787]: from='client.? 172.21.15.142:0/3136412482' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:26:47.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:26:46 smithi162 ceph-mon[33787]: pgmap v60: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:26:47.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:26:46 smithi142 ceph-mon[29958]: pgmap v60: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:26:47.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:26:46 smithi142 ceph-mon[37372]: pgmap v60: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:26:47.547 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 06 14:26:47 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-iscsi-iscsi-a[59853]: debug there is no tcmu-runner data available 2024-09-06T14:26:48.483 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph health --format=json 2024-09-06T14:26:48.742 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:26:49.026 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:26:48 smithi142 ceph-mon[29958]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-06T14:26:49.026 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:26:48 smithi142 ceph-mon[29958]: pgmap v61: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:26:49.026 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:26:48 smithi142 ceph-mon[37372]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-06T14:26:49.026 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:26:48 smithi142 ceph-mon[37372]: pgmap v61: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:26:49.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:26:48 smithi162 ceph-mon[33787]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-06T14:26:49.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:26:48 smithi162 ceph-mon[33787]: pgmap v61: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:26:49.222 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:49.215+0000 7ff80882b640 1 -- 172.21.15.142:0/806862981 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff8041038a0 msgr2=0x7ff804103c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:26:49.223 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:49.215+0000 7ff80882b640 1 --2- 172.21.15.142:0/806862981 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff8041038a0 0x7ff804103c80 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ff7f4009860 tx=0x7ff7f402a340 comp rx=0 tx=0).stop 2024-09-06T14:26:49.223 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:49.215+0000 7ff80882b640 1 -- 172.21.15.142:0/806862981 shutdown_connections 2024-09-06T14:26:49.223 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:49.215+0000 7ff80882b640 1 --2- 172.21.15.142:0/806862981 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff80410ce50 0x7ff80410f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:49.223 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:49.215+0000 7ff80882b640 1 --2- 172.21.15.142:0/806862981 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff8041042a0 0x7ff80410c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:49.223 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:49.215+0000 7ff80882b640 1 --2- 172.21.15.142:0/806862981 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff8041038a0 0x7ff804103c80 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:49.223 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:49.215+0000 7ff80882b640 1 -- 172.21.15.142:0/806862981 >> 172.21.15.142:0/806862981 conn(0x7ff8040fd430 msgr2=0x7ff8040ff850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:26:49.223 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:49.215+0000 7ff80882b640 1 -- 172.21.15.142:0/806862981 shutdown_connections 2024-09-06T14:26:49.223 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:49.215+0000 7ff80882b640 1 -- 172.21.15.142:0/806862981 wait complete. 2024-09-06T14:26:49.223 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:49.216+0000 7ff80882b640 1 Processor -- start 2024-09-06T14:26:49.223 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:49.216+0000 7ff80882b640 1 -- start start 2024-09-06T14:26:49.223 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:49.217+0000 7ff80882b640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff8041038a0 0x7ff8041986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:26:49.223 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:49.217+0000 7ff80882b640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff8041042a0 0x7ff804198bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:26:49.223 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:49.217+0000 7ff80882b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff80410ce50 0x7ff80419df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:26:49.224 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:49.217+0000 7ff80882b640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7ff8041121a0 con 0x7ff8041038a0 2024-09-06T14:26:49.224 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:49.217+0000 7ff80882b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff804112020 con 0x7ff80410ce50 2024-09-06T14:26:49.224 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:49.217+0000 7ff80882b640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7ff804112320 con 0x7ff8041042a0 2024-09-06T14:26:49.224 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:49.217+0000 7ff801d9b640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff8041042a0 0x7ff804198bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:26:49.224 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:49.217+0000 7ff80259c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff8041038a0 0x7ff8041986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:26:49.224 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:49.217+0000 7ff801d9b640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff8041042a0 0x7ff804198bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:33478/0 (socket says 172.21.15.142:33478) 2024-09-06T14:26:49.224 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:49.217+0000 7ff801d9b640 1 -- 172.21.15.142:0/1987071086 learned_addr learned my addr 172.21.15.142:0/1987071086 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:26:49.224 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:49.217+0000 7ff802d9d640 1 --2- 172.21.15.142:0/1987071086 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff80410ce50 0x7ff80419df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:26:49.224 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:49.217+0000 7ff801d9b640 1 -- 172.21.15.142:0/1987071086 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff80410ce50 msgr2=0x7ff80419df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:26:49.224 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:49.217+0000 7ff801d9b640 1 --2- 172.21.15.142:0/1987071086 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff80410ce50 0x7ff80419df90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:49.224 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:49.217+0000 7ff801d9b640 1 -- 172.21.15.142:0/1987071086 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff8041038a0 msgr2=0x7ff8041986b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:26:49.224 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:49.217+0000 7ff801d9b640 1 --2- 172.21.15.142:0/1987071086 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff8041038a0 0x7ff8041986b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:49.225 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:49.217+0000 7ff801d9b640 1 -- 172.21.15.142:0/1987071086 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff80419e760 con 0x7ff8041042a0 2024-09-06T14:26:49.225 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:49.217+0000 7ff80259c640 1 --2- 172.21.15.142:0/1987071086 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff8041038a0 0x7ff8041986b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-06T14:26:49.225 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:49.217+0000 7ff801d9b640 1 --2- 172.21.15.142:0/1987071086 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff8041042a0 0x7ff804198bf0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff7f800e9c0 tx=0x7ff7f800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:26:49.225 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:49.217+0000 7ff7f37fe640 1 -- 172.21.15.142:0/1987071086 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7f800cde0 con 0x7ff8041042a0 2024-09-06T14:26:49.225 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:49.217+0000 7ff7f37fe640 1 -- 172.21.15.142:0/1987071086 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7ff7f80035f0 con 0x7ff8041042a0 2024-09-06T14:26:49.225 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:49.217+0000 7ff80882b640 1 -- 172.21.15.142:0/1987071086 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff80419ea50 con 0x7ff8041042a0 2024-09-06T14:26:49.225 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:49.217+0000 7ff7f37fe640 1 -- 172.21.15.142:0/1987071086 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7f80044a0 con 0x7ff8041042a0 2024-09-06T14:26:49.225 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:49.217+0000 7ff80882b640 1 -- 172.21.15.142:0/1987071086 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8041a52e0 con 0x7ff8041042a0 2024-09-06T14:26:49.225 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:49.219+0000 7ff80882b640 1 -- 172.21.15.142:0/1987071086 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff804103c80 con 0x7ff8041042a0 2024-09-06T14:26:49.225 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:49.219+0000 7ff7f37fe640 1 -- 172.21.15.142:0/1987071086 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 22) ==== 99844+0+0 (secure 0 0 0) 0x7ff7f8004000 con 0x7ff8041042a0 2024-09-06T14:26:49.225 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:49.219+0000 7ff7f37fe640 1 --2- 172.21.15.142:0/1987071086 >> v2:172.21.15.142:6800/2944545612 conn(0x7ff7ec077620 0x7ff7ec079ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:26:49.225 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:49.220+0000 7ff7f37fe640 1 -- 172.21.15.142:0/1987071086 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(64..64 src has 1..64) ==== 6130+0+0 (secure 0 0 0) 0x7ff7f8099c90 con 0x7ff8041042a0 2024-09-06T14:26:49.225 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:49.220+0000 7ff80259c640 1 --2- 172.21.15.142:0/1987071086 >> v2:172.21.15.142:6800/2944545612 conn(0x7ff7ec077620 0x7ff7ec079ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:26:49.226 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:49.220+0000 7ff80259c640 1 --2- 172.21.15.142:0/1987071086 >> v2:172.21.15.142:6800/2944545612 conn(0x7ff7ec077620 0x7ff7ec079ae0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff7f402a930 tx=0x7ff7f40058d0 comp rx=0 tx=0).ready entity=mgr.14601 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:26:49.226 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:49.224+0000 7ff7f37fe640 1 -- 172.21.15.142:0/1987071086 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7ff7f809f020 con 0x7ff8041042a0 2024-09-06T14:26:49.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:49.435+0000 7ff80882b640 1 -- 172.21.15.142:0/1987071086 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7ff804107ec0 con 0x7ff8041042a0 2024-09-06T14:26:49.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:49.435+0000 7ff7f37fe640 1 -- 172.21.15.142:0/1987071086 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+212 (secure 0 0 0) 0x7ff7f8062dd0 con 0x7ff8041042a0 2024-09-06T14:26:49.433 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:26:49.433 INFO:teuthology.orchestra.run.smithi142.stdout:{"status":"HEALTH_WARN","checks":{"MGR_MODULE_DEPENDENCY":{"severity":"HEALTH_WARN","summary":{"message":"Module 'volumes' has failed dependency: No module named 'ceph.fs'","count":1},"muted":false}},"mutes":[]} 2024-09-06T14:26:49.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:49.438+0000 7ff80882b640 1 -- 172.21.15.142:0/1987071086 >> v2:172.21.15.142:6800/2944545612 conn(0x7ff7ec077620 msgr2=0x7ff7ec079ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:26:49.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:49.438+0000 7ff80882b640 1 --2- 172.21.15.142:0/1987071086 >> v2:172.21.15.142:6800/2944545612 conn(0x7ff7ec077620 0x7ff7ec079ae0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff7f402a930 tx=0x7ff7f40058d0 comp rx=0 tx=0).stop 2024-09-06T14:26:49.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:49.438+0000 7ff80882b640 1 -- 172.21.15.142:0/1987071086 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff8041042a0 msgr2=0x7ff804198bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:26:49.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:49.438+0000 7ff80882b640 1 --2- 172.21.15.142:0/1987071086 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff8041042a0 0x7ff804198bf0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff7f800e9c0 tx=0x7ff7f800ee80 comp rx=0 tx=0).stop 2024-09-06T14:26:49.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:49.438+0000 7ff80882b640 1 -- 172.21.15.142:0/1987071086 shutdown_connections 2024-09-06T14:26:49.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:49.438+0000 7ff80882b640 1 --2- 172.21.15.142:0/1987071086 >> v2:172.21.15.142:6800/2944545612 conn(0x7ff7ec077620 0x7ff7ec079ae0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:49.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:49.438+0000 7ff80882b640 1 --2- 172.21.15.142:0/1987071086 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff80410ce50 0x7ff80419df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:49.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:49.438+0000 7ff80882b640 1 --2- 172.21.15.142:0/1987071086 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff8041042a0 0x7ff804198bf0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:49.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:49.438+0000 7ff80882b640 1 --2- 172.21.15.142:0/1987071086 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff8041038a0 0x7ff8041986b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:49.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:49.438+0000 7ff80882b640 1 -- 172.21.15.142:0/1987071086 >> 172.21.15.142:0/1987071086 conn(0x7ff8040fd430 msgr2=0x7ff8040fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:26:49.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:49.438+0000 7ff80882b640 1 -- 172.21.15.142:0/1987071086 shutdown_connections 2024-09-06T14:26:49.436 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:49.438+0000 7ff80882b640 1 -- 172.21.15.142:0/1987071086 wait complete. 2024-09-06T14:26:50.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:26:49 smithi162 ceph-mon[33787]: from='client.? 172.21.15.142:0/1987071086' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:26:50.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:26:49 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/1987071086' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:26:50.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:26:49 smithi142 ceph-mon[37372]: from='client.? 172.21.15.142:0/1987071086' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:26:51.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:26:50 smithi162 ceph-mon[33787]: pgmap v62: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:26:51.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:26:50 smithi142 ceph-mon[29958]: pgmap v62: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:26:51.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:26:50 smithi142 ceph-mon[37372]: pgmap v62: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:26:53.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:26:52 smithi162 ceph-mon[33787]: pgmap v63: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:26:53.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:26:52 smithi142 ceph-mon[29958]: pgmap v63: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:26:53.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:26:52 smithi142 ceph-mon[37372]: pgmap v63: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:26:53.348 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph health --format=json 2024-09-06T14:26:53.614 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:26:54.156 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:54.129+0000 7f5a5c48d640 1 -- 172.21.15.142:0/2929879178 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5a54108e70 msgr2=0x7f5a5410f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:26:54.156 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:54.129+0000 7f5a5c48d640 1 --2- 172.21.15.142:0/2929879178 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5a54108e70 0x7f5a5410f670 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f5a44009900 tx=0x7f5a4402a410 comp rx=0 tx=0).stop 2024-09-06T14:26:54.156 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:54.129+0000 7f5a5c48d640 1 -- 172.21.15.142:0/2929879178 shutdown_connections 2024-09-06T14:26:54.156 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:54.129+0000 7f5a5c48d640 1 --2- 172.21.15.142:0/2929879178 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5a54108e70 0x7f5a5410f670 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:54.156 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:54.129+0000 7f5a5c48d640 1 --2- 172.21.15.142:0/2929879178 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5a541084f0 0x7f5a54108930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:54.156 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:54.129+0000 7f5a5c48d640 1 --2- 172.21.15.142:0/2929879178 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5a54102ae0 0x7f5a54102ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:54.156 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:54.129+0000 7f5a5c48d640 1 -- 172.21.15.142:0/2929879178 >> 172.21.15.142:0/2929879178 conn(0x7f5a540fe530 msgr2=0x7f5a54100950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:26:54.156 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:54.129+0000 7f5a5c48d640 1 -- 172.21.15.142:0/2929879178 shutdown_connections 2024-09-06T14:26:54.156 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:54.129+0000 7f5a5c48d640 1 -- 172.21.15.142:0/2929879178 wait complete. 2024-09-06T14:26:54.156 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:54.130+0000 7f5a5c48d640 1 Processor -- start 2024-09-06T14:26:54.157 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:54.130+0000 7f5a5c48d640 1 -- start start 2024-09-06T14:26:54.157 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:54.130+0000 7f5a5c48d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5a54102ae0 0x7f5a54198660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:26:54.157 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:54.130+0000 7f5a5c48d640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5a541084f0 0x7f5a54198ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:26:54.157 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:54.130+0000 7f5a5c48d640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5a54108e70 0x7f5a5419df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:26:54.157 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:54.130+0000 7f5a5c48d640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f5a54112250 con 0x7f5a541084f0 2024-09-06T14:26:54.157 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:54.130+0000 7f5a5c48d640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5a541120d0 con 0x7f5a54102ae0 2024-09-06T14:26:54.157 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:54.130+0000 7f5a5c48d640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7f5a541123d0 con 0x7f5a54108e70 2024-09-06T14:26:54.157 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:54.130+0000 7f5a59a28640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5a541084f0 0x7f5a54198ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:26:54.157 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:54.130+0000 7f5a5aa2a640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5a54108e70 0x7f5a5419df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:26:54.157 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:54.130+0000 7f5a5aa2a640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5a54108e70 0x7f5a5419df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:33490/0 (socket says 172.21.15.142:33490) 2024-09-06T14:26:54.157 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:54.130+0000 7f5a59a28640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5a541084f0 0x7f5a54198ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:50756/0 (socket says 172.21.15.142:50756) 2024-09-06T14:26:54.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:54.130+0000 7f5a5aa2a640 1 -- 172.21.15.142:0/4238167569 learned_addr learned my addr 172.21.15.142:0/4238167569 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:26:54.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:54.130+0000 7f5a5a229640 1 --2- 172.21.15.142:0/4238167569 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5a54102ae0 0x7f5a54198660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:26:54.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:54.131+0000 7f5a5aa2a640 1 -- 172.21.15.142:0/4238167569 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5a54102ae0 msgr2=0x7f5a54198660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:26:54.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:54.131+0000 7f5a5aa2a640 1 --2- 172.21.15.142:0/4238167569 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5a54102ae0 0x7f5a54198660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:54.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:54.131+0000 7f5a5aa2a640 1 -- 172.21.15.142:0/4238167569 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5a541084f0 msgr2=0x7f5a54198ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:26:54.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:54.131+0000 7f5a5aa2a640 1 --2- 172.21.15.142:0/4238167569 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5a541084f0 0x7f5a54198ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:54.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:54.131+0000 7f5a5aa2a640 1 -- 172.21.15.142:0/4238167569 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a5419e710 con 0x7f5a54108e70 2024-09-06T14:26:54.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:54.131+0000 7f5a5aa2a640 1 --2- 172.21.15.142:0/4238167569 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5a54108e70 0x7f5a5419df40 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5a4402a930 tx=0x7f5a44005000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:26:54.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:54.131+0000 7f5a537fe640 1 -- 172.21.15.142:0/4238167569 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a4403f070 con 0x7f5a54108e70 2024-09-06T14:26:54.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:54.131+0000 7f5a5c48d640 1 -- 172.21.15.142:0/4238167569 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a5419e9a0 con 0x7f5a54108e70 2024-09-06T14:26:54.159 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:54.131+0000 7f5a537fe640 1 -- 172.21.15.142:0/4238167569 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7f5a44005600 con 0x7f5a54108e70 2024-09-06T14:26:54.159 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:54.131+0000 7f5a537fe640 1 -- 172.21.15.142:0/4238167569 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a44035030 con 0x7f5a54108e70 2024-09-06T14:26:54.159 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:54.131+0000 7f5a5c48d640 1 -- 172.21.15.142:0/4238167569 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a541a50c0 con 0x7f5a54108e70 2024-09-06T14:26:54.159 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:54.132+0000 7f5a5c48d640 1 -- 172.21.15.142:0/4238167569 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a54102ec0 con 0x7f5a54108e70 2024-09-06T14:26:54.159 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:54.132+0000 7f5a537fe640 1 -- 172.21.15.142:0/4238167569 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 22) ==== 99844+0+0 (secure 0 0 0) 0x7f5a44042070 con 0x7f5a54108e70 2024-09-06T14:26:54.159 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:54.133+0000 7f5a537fe640 1 --2- 172.21.15.142:0/4238167569 >> v2:172.21.15.142:6800/2944545612 conn(0x7f5a3c077620 0x7f5a3c079ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:26:54.159 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:54.133+0000 7f5a537fe640 1 -- 172.21.15.142:0/4238167569 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(64..64 src has 1..64) ==== 6130+0+0 (secure 0 0 0) 0x7f5a440ba130 con 0x7f5a54108e70 2024-09-06T14:26:54.159 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:54.133+0000 7f5a5a229640 1 --2- 172.21.15.142:0/4238167569 >> v2:172.21.15.142:6800/2944545612 conn(0x7f5a3c077620 0x7f5a3c079ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:26:54.159 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:54.133+0000 7f5a5a229640 1 --2- 172.21.15.142:0/4238167569 >> v2:172.21.15.142:6800/2944545612 conn(0x7f5a3c077620 0x7f5a3c079ae0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f5a4c0099a0 tx=0x7f5a4c009500 comp rx=0 tx=0).ready entity=mgr.14601 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:26:54.159 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:54.137+0000 7f5a537fe640 1 -- 172.21.15.142:0/4238167569 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7f5a44087360 con 0x7f5a54108e70 2024-09-06T14:26:54.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:54.323+0000 7f5a5c48d640 1 -- 172.21.15.142:0/4238167569 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f5a541a5290 con 0x7f5a54108e70 2024-09-06T14:26:54.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:54.323+0000 7f5a537fe640 1 -- 172.21.15.142:0/4238167569 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+212 (secure 0 0 0) 0x7f5a4408bd80 con 0x7f5a54108e70 2024-09-06T14:26:54.321 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:26:54.321 INFO:teuthology.orchestra.run.smithi142.stdout:{"status":"HEALTH_WARN","checks":{"MGR_MODULE_DEPENDENCY":{"severity":"HEALTH_WARN","summary":{"message":"Module 'volumes' has failed dependency: No module named 'ceph.fs'","count":1},"muted":false}},"mutes":[]} 2024-09-06T14:26:54.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:54.325+0000 7f5a5c48d640 1 -- 172.21.15.142:0/4238167569 >> v2:172.21.15.142:6800/2944545612 conn(0x7f5a3c077620 msgr2=0x7f5a3c079ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:26:54.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:54.325+0000 7f5a5c48d640 1 --2- 172.21.15.142:0/4238167569 >> v2:172.21.15.142:6800/2944545612 conn(0x7f5a3c077620 0x7f5a3c079ae0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f5a4c0099a0 tx=0x7f5a4c009500 comp rx=0 tx=0).stop 2024-09-06T14:26:54.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:54.325+0000 7f5a5c48d640 1 -- 172.21.15.142:0/4238167569 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5a54108e70 msgr2=0x7f5a5419df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:26:54.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:54.325+0000 7f5a5c48d640 1 --2- 172.21.15.142:0/4238167569 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5a54108e70 0x7f5a5419df40 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5a4402a930 tx=0x7f5a44005000 comp rx=0 tx=0).stop 2024-09-06T14:26:54.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:54.325+0000 7f5a5c48d640 1 -- 172.21.15.142:0/4238167569 shutdown_connections 2024-09-06T14:26:54.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:54.325+0000 7f5a5c48d640 1 --2- 172.21.15.142:0/4238167569 >> v2:172.21.15.142:6800/2944545612 conn(0x7f5a3c077620 0x7f5a3c079ae0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:54.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:54.325+0000 7f5a5c48d640 1 --2- 172.21.15.142:0/4238167569 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5a54108e70 0x7f5a5419df40 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:54.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:54.325+0000 7f5a5c48d640 1 --2- 172.21.15.142:0/4238167569 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5a541084f0 0x7f5a54198ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:54.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:54.325+0000 7f5a5c48d640 1 --2- 172.21.15.142:0/4238167569 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5a54102ae0 0x7f5a54198660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:54.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:54.325+0000 7f5a5c48d640 1 -- 172.21.15.142:0/4238167569 >> 172.21.15.142:0/4238167569 conn(0x7f5a540fe530 msgr2=0x7f5a540ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:26:54.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:54.326+0000 7f5a5c48d640 1 -- 172.21.15.142:0/4238167569 shutdown_connections 2024-09-06T14:26:54.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:54.326+0000 7f5a5c48d640 1 -- 172.21.15.142:0/4238167569 wait complete. 2024-09-06T14:26:54.709 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:26:54 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: ::ffff:172.21.15.162 - - [06/Sep/2024:14:26:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-06T14:26:55.045 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:26:54 smithi142 ceph-mon[29958]: pgmap v64: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:26:55.045 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:26:54 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/4238167569' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:26:55.045 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:26:54 smithi142 ceph-mon[37372]: pgmap v64: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:26:55.045 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:26:54 smithi142 ceph-mon[37372]: from='client.? 172.21.15.142:0/4238167569' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:26:55.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:26:54 smithi162 ceph-mon[33787]: pgmap v64: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:26:55.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:26:54 smithi162 ceph-mon[33787]: from='client.? 172.21.15.142:0/4238167569' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:26:57.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:26:56 smithi162 ceph-mon[33787]: pgmap v65: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:26:57.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:26:56 smithi162 ceph-mon[33787]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-06T14:26:57.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:26:56 smithi142 ceph-mon[29958]: pgmap v65: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:26:57.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:26:56 smithi142 ceph-mon[29958]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-06T14:26:57.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:26:56 smithi142 ceph-mon[37372]: pgmap v65: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:26:57.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:26:56 smithi142 ceph-mon[37372]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-06T14:26:57.547 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 06 14:26:57 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-iscsi-iscsi-a[59853]: debug there is no tcmu-runner data available 2024-09-06T14:26:58.078 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph health --format=json 2024-09-06T14:26:58.341 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:26:58.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:58.826+0000 7f7f6bba9640 1 -- 172.21.15.142:0/1538290827 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7f641084f0 msgr2=0x7f7f64108930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:26:58.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:58.826+0000 7f7f6bba9640 1 --2- 172.21.15.142:0/1538290827 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7f641084f0 0x7f7f64108930 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f7f58009900 tx=0x7f7f5802a410 comp rx=0 tx=0).stop 2024-09-06T14:26:58.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:58.826+0000 7f7f6bba9640 1 -- 172.21.15.142:0/1538290827 shutdown_connections 2024-09-06T14:26:58.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:58.826+0000 7f7f6bba9640 1 --2- 172.21.15.142:0/1538290827 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7f64108e70 0x7f7f6410f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:58.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:58.826+0000 7f7f6bba9640 1 --2- 172.21.15.142:0/1538290827 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7f641084f0 0x7f7f64108930 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:58.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:58.826+0000 7f7f6bba9640 1 --2- 172.21.15.142:0/1538290827 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7f64102ae0 0x7f7f64102ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:58.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:58.826+0000 7f7f6bba9640 1 -- 172.21.15.142:0/1538290827 >> 172.21.15.142:0/1538290827 conn(0x7f7f640fe530 msgr2=0x7f7f64100950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:26:58.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:58.826+0000 7f7f6bba9640 1 -- 172.21.15.142:0/1538290827 shutdown_connections 2024-09-06T14:26:58.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:58.826+0000 7f7f6bba9640 1 -- 172.21.15.142:0/1538290827 wait complete. 2024-09-06T14:26:58.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:58.826+0000 7f7f6bba9640 1 Processor -- start 2024-09-06T14:26:58.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:58.827+0000 7f7f6bba9640 1 -- start start 2024-09-06T14:26:58.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:58.827+0000 7f7f6bba9640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7f64102ae0 0x7f7f64075200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:26:58.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:58.827+0000 7f7f6bba9640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7f641084f0 0x7f7f64075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:26:58.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:58.827+0000 7f7f6bba9640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7f64108e70 0x7f7f64075c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:26:58.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:58.827+0000 7f7f6bba9640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f7f64112240 con 0x7f7f641084f0 2024-09-06T14:26:58.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:58.827+0000 7f7f6bba9640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7f641120c0 con 0x7f7f64102ae0 2024-09-06T14:26:58.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:58.827+0000 7f7f6bba9640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7f7f641123c0 con 0x7f7f64108e70 2024-09-06T14:26:58.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:58.827+0000 7f7f69144640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7f641084f0 0x7f7f64075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:26:58.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:58.827+0000 7f7f6a146640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7f64108e70 0x7f7f64075c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:26:58.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:58.827+0000 7f7f69144640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7f641084f0 0x7f7f64075740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:44118/0 (socket says 172.21.15.142:44118) 2024-09-06T14:26:58.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:58.827+0000 7f7f69144640 1 -- 172.21.15.142:0/1451187559 learned_addr learned my addr 172.21.15.142:0/1451187559 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:26:58.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:58.827+0000 7f7f69945640 1 --2- 172.21.15.142:0/1451187559 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7f64102ae0 0x7f7f64075200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:26:58.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:58.827+0000 7f7f6a146640 1 -- 172.21.15.142:0/1451187559 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7f64102ae0 msgr2=0x7f7f64075200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:26:58.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:58.827+0000 7f7f6a146640 1 --2- 172.21.15.142:0/1451187559 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7f64102ae0 0x7f7f64075200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:58.836 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:58.827+0000 7f7f6a146640 1 -- 172.21.15.142:0/1451187559 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7f641084f0 msgr2=0x7f7f64075740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:26:58.836 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:58.827+0000 7f7f6a146640 1 --2- 172.21.15.142:0/1451187559 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7f641084f0 0x7f7f64075740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:58.836 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:58.827+0000 7f7f6a146640 1 -- 172.21.15.142:0/1451187559 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f641a6b50 con 0x7f7f64108e70 2024-09-06T14:26:58.836 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:58.827+0000 7f7f69945640 1 --2- 172.21.15.142:0/1451187559 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7f64102ae0 0x7f7f64075200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-06T14:26:58.836 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:58.827+0000 7f7f69144640 1 --2- 172.21.15.142:0/1451187559 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7f641084f0 0x7f7f64075740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-06T14:26:58.836 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:58.827+0000 7f7f6a146640 1 --2- 172.21.15.142:0/1451187559 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7f64108e70 0x7f7f64075c80 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f7f5c00ec50 tx=0x7f7f5c00c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:26:58.836 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:58.827+0000 7f7f56ffd640 1 -- 172.21.15.142:0/1451187559 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f5c003b70 con 0x7f7f64108e70 2024-09-06T14:26:58.836 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:58.827+0000 7f7f56ffd640 1 -- 172.21.15.142:0/1451187559 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7f7f5c003d10 con 0x7f7f64108e70 2024-09-06T14:26:58.836 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:58.828+0000 7f7f6bba9640 1 -- 172.21.15.142:0/1451187559 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f641a6e40 con 0x7f7f64108e70 2024-09-06T14:26:58.836 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:58.828+0000 7f7f56ffd640 1 -- 172.21.15.142:0/1451187559 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f5c010650 con 0x7f7f64108e70 2024-09-06T14:26:58.836 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:58.828+0000 7f7f6bba9640 1 -- 172.21.15.142:0/1451187559 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f641a72a0 con 0x7f7f64108e70 2024-09-06T14:26:58.836 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:58.829+0000 7f7f6bba9640 1 -- 172.21.15.142:0/1451187559 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f64102ec0 con 0x7f7f64108e70 2024-09-06T14:26:58.836 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:58.829+0000 7f7f56ffd640 1 -- 172.21.15.142:0/1451187559 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 22) ==== 99844+0+0 (secure 0 0 0) 0x7f7f5c004020 con 0x7f7f64108e70 2024-09-06T14:26:58.836 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:58.830+0000 7f7f56ffd640 1 --2- 172.21.15.142:0/1451187559 >> v2:172.21.15.142:6800/2944545612 conn(0x7f7f50077620 0x7f7f50079ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:26:58.837 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:58.830+0000 7f7f56ffd640 1 -- 172.21.15.142:0/1451187559 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(64..64 src has 1..64) ==== 6130+0+0 (secure 0 0 0) 0x7f7f5c099ea0 con 0x7f7f64108e70 2024-09-06T14:26:58.837 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:58.830+0000 7f7f69945640 1 --2- 172.21.15.142:0/1451187559 >> v2:172.21.15.142:6800/2944545612 conn(0x7f7f50077620 0x7f7f50079ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:26:58.837 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:58.830+0000 7f7f69945640 1 --2- 172.21.15.142:0/1451187559 >> v2:172.21.15.142:6800/2944545612 conn(0x7f7f50077620 0x7f7f50079ae0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f7f600058c0 tx=0x7f7f60005d30 comp rx=0 tx=0).ready entity=mgr.14601 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:26:58.837 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:58.834+0000 7f7f56ffd640 1 -- 172.21.15.142:0/1451187559 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7f7f5c062f60 con 0x7f7f64108e70 2024-09-06T14:26:59.020 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:59.023+0000 7f7f6bba9640 1 -- 172.21.15.142:0/1451187559 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f7f6410ac50 con 0x7f7f64108e70 2024-09-06T14:26:59.020 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:26:58 smithi142 ceph-mon[29958]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-06T14:26:59.020 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:26:58 smithi142 ceph-mon[29958]: pgmap v66: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:26:59.021 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:26:58 smithi142 ceph-mon[37372]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-06T14:26:59.021 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:26:58 smithi142 ceph-mon[37372]: pgmap v66: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:26:59.021 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:59.023+0000 7f7f56ffd640 1 -- 172.21.15.142:0/1451187559 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+212 (secure 0 0 0) 0x7f7f5c0670d0 con 0x7f7f64108e70 2024-09-06T14:26:59.021 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:26:59.021 INFO:teuthology.orchestra.run.smithi142.stdout:{"status":"HEALTH_WARN","checks":{"MGR_MODULE_DEPENDENCY":{"severity":"HEALTH_WARN","summary":{"message":"Module 'volumes' has failed dependency: No module named 'ceph.fs'","count":1},"muted":false}},"mutes":[]} 2024-09-06T14:26:59.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:59.025+0000 7f7f6bba9640 1 -- 172.21.15.142:0/1451187559 >> v2:172.21.15.142:6800/2944545612 conn(0x7f7f50077620 msgr2=0x7f7f50079ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:26:59.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:59.025+0000 7f7f6bba9640 1 --2- 172.21.15.142:0/1451187559 >> v2:172.21.15.142:6800/2944545612 conn(0x7f7f50077620 0x7f7f50079ae0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f7f600058c0 tx=0x7f7f60005d30 comp rx=0 tx=0).stop 2024-09-06T14:26:59.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:59.025+0000 7f7f6bba9640 1 -- 172.21.15.142:0/1451187559 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7f64108e70 msgr2=0x7f7f64075c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:26:59.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:59.025+0000 7f7f6bba9640 1 --2- 172.21.15.142:0/1451187559 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7f64108e70 0x7f7f64075c80 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f7f5c00ec50 tx=0x7f7f5c00c6a0 comp rx=0 tx=0).stop 2024-09-06T14:26:59.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:59.025+0000 7f7f6bba9640 1 -- 172.21.15.142:0/1451187559 shutdown_connections 2024-09-06T14:26:59.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:59.025+0000 7f7f6bba9640 1 --2- 172.21.15.142:0/1451187559 >> v2:172.21.15.142:6800/2944545612 conn(0x7f7f50077620 0x7f7f50079ae0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:59.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:59.025+0000 7f7f6bba9640 1 --2- 172.21.15.142:0/1451187559 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7f64108e70 0x7f7f64075c80 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:59.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:59.025+0000 7f7f6bba9640 1 --2- 172.21.15.142:0/1451187559 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7f641084f0 0x7f7f64075740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:59.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:59.025+0000 7f7f6bba9640 1 --2- 172.21.15.142:0/1451187559 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7f64102ae0 0x7f7f64075200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:26:59.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:59.025+0000 7f7f6bba9640 1 -- 172.21.15.142:0/1451187559 >> 172.21.15.142:0/1451187559 conn(0x7f7f640fe530 msgr2=0x7f7f64100920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:26:59.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:59.026+0000 7f7f6bba9640 1 -- 172.21.15.142:0/1451187559 shutdown_connections 2024-09-06T14:26:59.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:26:59.026+0000 7f7f6bba9640 1 -- 172.21.15.142:0/1451187559 wait complete. 2024-09-06T14:26:59.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:26:58 smithi162 ceph-mon[33787]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-06T14:26:59.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:26:58 smithi162 ceph-mon[33787]: pgmap v66: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:27:00.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:26:59 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/1451187559' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:27:00.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:26:59 smithi142 ceph-mon[37372]: from='client.? 172.21.15.142:0/1451187559' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:27:00.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:26:59 smithi162 ceph-mon[33787]: from='client.? 172.21.15.142:0/1451187559' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:27:01.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:27:00 smithi142 ceph-mon[29958]: pgmap v67: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:27:01.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:27:00 smithi142 ceph-mon[37372]: pgmap v67: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:27:01.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:27:00 smithi162 ceph-mon[33787]: pgmap v67: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:27:02.778 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph health --format=json 2024-09-06T14:27:03.038 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:27:03.159 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:27:02 smithi142 ceph-mon[29958]: pgmap v68: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:27:03.159 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:27:02 smithi142 ceph-mon[37372]: pgmap v68: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:27:03.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:27:02 smithi162 ceph-mon[33787]: pgmap v68: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:27:03.520 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:03.510+0000 7fbbfce32640 1 -- 172.21.15.142:0/2469784918 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbbf81019d0 msgr2=0x7fbbf8101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:27:03.520 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:03.510+0000 7fbbfce32640 1 --2- 172.21.15.142:0/2469784918 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbbf81019d0 0x7fbbf8101db0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fbbe8009900 tx=0x7fbbe802a410 comp rx=0 tx=0).stop 2024-09-06T14:27:03.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:03.510+0000 7fbbfce32640 1 -- 172.21.15.142:0/2469784918 shutdown_connections 2024-09-06T14:27:03.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:03.510+0000 7fbbfce32640 1 --2- 172.21.15.142:0/2469784918 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbbf8107d60 0x7fbbf810e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:03.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:03.510+0000 7fbbfce32640 1 --2- 172.21.15.142:0/2469784918 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbbf81073e0 0x7fbbf8107820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:03.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:03.510+0000 7fbbfce32640 1 --2- 172.21.15.142:0/2469784918 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbbf81019d0 0x7fbbf8101db0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:03.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:03.510+0000 7fbbfce32640 1 -- 172.21.15.142:0/2469784918 >> 172.21.15.142:0/2469784918 conn(0x7fbbf80fd420 msgr2=0x7fbbf80ff840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:27:03.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:03.510+0000 7fbbfce32640 1 -- 172.21.15.142:0/2469784918 shutdown_connections 2024-09-06T14:27:03.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:03.510+0000 7fbbfce32640 1 -- 172.21.15.142:0/2469784918 wait complete. 2024-09-06T14:27:03.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:03.511+0000 7fbbfce32640 1 Processor -- start 2024-09-06T14:27:03.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:03.511+0000 7fbbfce32640 1 -- start start 2024-09-06T14:27:03.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:03.511+0000 7fbbfce32640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbbf81019d0 0x7fbbf819a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:27:03.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:03.511+0000 7fbbfce32640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbbf81073e0 0x7fbbf819ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:27:03.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:03.511+0000 7fbbfce32640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbbf8107d60 0x7fbbf81a01b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:27:03.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:03.511+0000 7fbbfce32640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fbbf8111130 con 0x7fbbf81019d0 2024-09-06T14:27:03.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:03.511+0000 7fbbfce32640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fbbf8110fb0 con 0x7fbbf8107d60 2024-09-06T14:27:03.522 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:03.511+0000 7fbbfce32640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7fbbf81112b0 con 0x7fbbf81073e0 2024-09-06T14:27:03.522 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:03.511+0000 7fbbf659c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbbf81019d0 0x7fbbf819a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:27:03.522 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:03.511+0000 7fbbf5d9b640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbbf81073e0 0x7fbbf819ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:27:03.522 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:03.511+0000 7fbbf5d9b640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbbf81073e0 0x7fbbf819ae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:42330/0 (socket says 172.21.15.142:42330) 2024-09-06T14:27:03.522 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:03.511+0000 7fbbf5d9b640 1 -- 172.21.15.142:0/920382429 learned_addr learned my addr 172.21.15.142:0/920382429 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:27:03.522 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:03.512+0000 7fbbf6d9d640 1 --2- 172.21.15.142:0/920382429 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbbf8107d60 0x7fbbf81a01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:27:03.522 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:03.512+0000 7fbbf5d9b640 1 -- 172.21.15.142:0/920382429 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbbf8107d60 msgr2=0x7fbbf81a01b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:27:03.522 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:03.512+0000 7fbbf5d9b640 1 --2- 172.21.15.142:0/920382429 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbbf8107d60 0x7fbbf81a01b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:03.522 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:03.512+0000 7fbbf5d9b640 1 -- 172.21.15.142:0/920382429 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbbf81019d0 msgr2=0x7fbbf819a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:27:03.522 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:03.512+0000 7fbbf5d9b640 1 --2- 172.21.15.142:0/920382429 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbbf81019d0 0x7fbbf819a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:03.522 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:03.512+0000 7fbbf5d9b640 1 -- 172.21.15.142:0/920382429 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbf81a0980 con 0x7fbbf81073e0 2024-09-06T14:27:03.522 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:03.512+0000 7fbbf659c640 1 --2- 172.21.15.142:0/920382429 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbbf81019d0 0x7fbbf819a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-06T14:27:03.522 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:03.512+0000 7fbbf5d9b640 1 --2- 172.21.15.142:0/920382429 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbbf81073e0 0x7fbbf819ae10 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fbbec00d770 tx=0x7fbbec00dc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:27:03.522 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:03.512+0000 7fbbe77fe640 1 -- 172.21.15.142:0/920382429 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbec017070 con 0x7fbbf81073e0 2024-09-06T14:27:03.523 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:03.512+0000 7fbbe77fe640 1 -- 172.21.15.142:0/920382429 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7fbbec0054a0 con 0x7fbbf81073e0 2024-09-06T14:27:03.523 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:03.512+0000 7fbbfce32640 1 -- 172.21.15.142:0/920382429 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbbf81a0c70 con 0x7fbbf81073e0 2024-09-06T14:27:03.523 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:03.512+0000 7fbbe77fe640 1 -- 172.21.15.142:0/920382429 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbec004c40 con 0x7fbbf81073e0 2024-09-06T14:27:03.523 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:03.512+0000 7fbbfce32640 1 -- 172.21.15.142:0/920382429 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fbbf81a7330 con 0x7fbbf81073e0 2024-09-06T14:27:03.523 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:03.513+0000 7fbbfce32640 1 -- 172.21.15.142:0/920382429 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbbf8101db0 con 0x7fbbf81073e0 2024-09-06T14:27:03.523 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:03.514+0000 7fbbe77fe640 1 -- 172.21.15.142:0/920382429 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 22) ==== 99844+0+0 (secure 0 0 0) 0x7fbbec004de0 con 0x7fbbf81073e0 2024-09-06T14:27:03.523 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:03.514+0000 7fbbe77fe640 1 --2- 172.21.15.142:0/920382429 >> v2:172.21.15.142:6800/2944545612 conn(0x7fbbe0077620 0x7fbbe0079ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:27:03.523 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:03.514+0000 7fbbe77fe640 1 -- 172.21.15.142:0/920382429 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(64..64 src has 1..64) ==== 6130+0+0 (secure 0 0 0) 0x7fbbec099380 con 0x7fbbf81073e0 2024-09-06T14:27:03.523 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:03.514+0000 7fbbf659c640 1 --2- 172.21.15.142:0/920382429 >> v2:172.21.15.142:6800/2944545612 conn(0x7fbbe0077620 0x7fbbe0079ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:27:03.523 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:03.515+0000 7fbbf659c640 1 --2- 172.21.15.142:0/920382429 >> v2:172.21.15.142:6800/2944545612 conn(0x7fbbe0077620 0x7fbbe0079ae0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fbbe8031440 tx=0x7fbbe8031850 comp rx=0 tx=0).ready entity=mgr.14601 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:27:03.523 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:03.518+0000 7fbbe77fe640 1 -- 172.21.15.142:0/920382429 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7fbbec066630 con 0x7fbbf81073e0 2024-09-06T14:27:03.705 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:03.708+0000 7fbbfce32640 1 -- 172.21.15.142:0/920382429 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fbbf81a7650 con 0x7fbbf81073e0 2024-09-06T14:27:03.705 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:03.708+0000 7fbbe77fe640 1 -- 172.21.15.142:0/920382429 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+212 (secure 0 0 0) 0x7fbbec06b050 con 0x7fbbf81073e0 2024-09-06T14:27:03.705 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:27:03.706 INFO:teuthology.orchestra.run.smithi142.stdout:{"status":"HEALTH_WARN","checks":{"MGR_MODULE_DEPENDENCY":{"severity":"HEALTH_WARN","summary":{"message":"Module 'volumes' has failed dependency: No module named 'ceph.fs'","count":1},"muted":false}},"mutes":[]} 2024-09-06T14:27:03.707 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:03.710+0000 7fbbfce32640 1 -- 172.21.15.142:0/920382429 >> v2:172.21.15.142:6800/2944545612 conn(0x7fbbe0077620 msgr2=0x7fbbe0079ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:27:03.707 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:03.710+0000 7fbbfce32640 1 --2- 172.21.15.142:0/920382429 >> v2:172.21.15.142:6800/2944545612 conn(0x7fbbe0077620 0x7fbbe0079ae0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fbbe8031440 tx=0x7fbbe8031850 comp rx=0 tx=0).stop 2024-09-06T14:27:03.707 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:03.710+0000 7fbbfce32640 1 -- 172.21.15.142:0/920382429 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbbf81073e0 msgr2=0x7fbbf819ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:27:03.708 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:03.710+0000 7fbbfce32640 1 --2- 172.21.15.142:0/920382429 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbbf81073e0 0x7fbbf819ae10 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fbbec00d770 tx=0x7fbbec00dc30 comp rx=0 tx=0).stop 2024-09-06T14:27:03.708 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:03.710+0000 7fbbfce32640 1 -- 172.21.15.142:0/920382429 shutdown_connections 2024-09-06T14:27:03.708 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:03.710+0000 7fbbfce32640 1 --2- 172.21.15.142:0/920382429 >> v2:172.21.15.142:6800/2944545612 conn(0x7fbbe0077620 0x7fbbe0079ae0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:03.708 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:03.710+0000 7fbbfce32640 1 --2- 172.21.15.142:0/920382429 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbbf8107d60 0x7fbbf81a01b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:03.708 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:03.710+0000 7fbbfce32640 1 --2- 172.21.15.142:0/920382429 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbbf81073e0 0x7fbbf819ae10 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:03.708 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:03.710+0000 7fbbfce32640 1 --2- 172.21.15.142:0/920382429 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbbf81019d0 0x7fbbf819a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:03.708 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:03.710+0000 7fbbfce32640 1 -- 172.21.15.142:0/920382429 >> 172.21.15.142:0/920382429 conn(0x7fbbf80fd420 msgr2=0x7fbbf806b380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:27:03.708 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:03.710+0000 7fbbfce32640 1 -- 172.21.15.142:0/920382429 shutdown_connections 2024-09-06T14:27:03.708 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:03.710+0000 7fbbfce32640 1 -- 172.21.15.142:0/920382429 wait complete. 2024-09-06T14:27:03.960 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:27:03 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/920382429' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:27:03.961 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:27:03 smithi142 ceph-mon[37372]: from='client.? 172.21.15.142:0/920382429' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:27:04.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:27:03 smithi162 ceph-mon[33787]: from='client.? 172.21.15.142:0/920382429' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:27:04.459 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:27:04 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: ::ffff:172.21.15.162 - - [06/Sep/2024:14:27:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-06T14:27:05.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:27:04 smithi142 ceph-mon[29958]: pgmap v69: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:27:05.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:27:04 smithi142 ceph-mon[37372]: pgmap v69: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:27:05.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:27:04 smithi162 ceph-mon[33787]: pgmap v69: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:27:07.132 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:27:06 smithi162 ceph-mon[33787]: pgmap v70: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:27:07.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:27:06 smithi142 ceph-mon[29958]: pgmap v70: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:27:07.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:27:06 smithi142 ceph-mon[37372]: pgmap v70: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:27:07.480 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph health --format=json 2024-09-06T14:27:07.547 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 06 14:27:07 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-iscsi-iscsi-a[59853]: debug there is no tcmu-runner data available 2024-09-06T14:27:07.740 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:27:08.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:08.269+0000 7f930a614640 1 -- 172.21.15.142:0/3631973331 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9304103a10 msgr2=0x7f9304103df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:27:08.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:08.269+0000 7f930a614640 1 --2- 172.21.15.142:0/3631973331 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9304103a10 0x7f9304103df0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f92f4009900 tx=0x7f92f402a410 comp rx=0 tx=0).stop 2024-09-06T14:27:08.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:08.269+0000 7f930a614640 1 -- 172.21.15.142:0/3631973331 shutdown_connections 2024-09-06T14:27:08.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:08.269+0000 7f930a614640 1 --2- 172.21.15.142:0/3631973331 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f930410ce70 0x7f930410f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:08.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:08.269+0000 7f930a614640 1 --2- 172.21.15.142:0/3631973331 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9304104330 0x7f930410c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:08.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:08.269+0000 7f930a614640 1 --2- 172.21.15.142:0/3631973331 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9304103a10 0x7f9304103df0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:08.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:08.269+0000 7f930a614640 1 -- 172.21.15.142:0/3631973331 >> 172.21.15.142:0/3631973331 conn(0x7f93040fd470 msgr2=0x7f93040ff890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:27:08.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:08.269+0000 7f930a614640 1 -- 172.21.15.142:0/3631973331 shutdown_connections 2024-09-06T14:27:08.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:08.269+0000 7f930a614640 1 -- 172.21.15.142:0/3631973331 wait complete. 2024-09-06T14:27:08.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:08.269+0000 7f930a614640 1 Processor -- start 2024-09-06T14:27:08.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:08.269+0000 7f930a614640 1 -- start start 2024-09-06T14:27:08.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:08.270+0000 7f930a614640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9304103a10 0x7f9304198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:27:08.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:08.270+0000 7f930a614640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9304104330 0x7f9304198c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:27:08.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:08.270+0000 7f930a614640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f930410ce70 0x7f930419e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:27:08.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:08.270+0000 7f930a614640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f93041121f0 con 0x7f9304104330 2024-09-06T14:27:08.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:08.270+0000 7f930a614640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f9304112070 con 0x7f930410ce70 2024-09-06T14:27:08.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:08.270+0000 7f930a614640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7f9304112370 con 0x7f9304103a10 2024-09-06T14:27:08.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:08.270+0000 7f9303fff640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9304103a10 0x7f9304198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:27:08.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:08.270+0000 7f93037fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9304104330 0x7f9304198c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:27:08.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:08.270+0000 7f93037fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9304104330 0x7f9304198c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:39284/0 (socket says 172.21.15.142:39284) 2024-09-06T14:27:08.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:08.270+0000 7f93037fe640 1 -- 172.21.15.142:0/3542072778 learned_addr learned my addr 172.21.15.142:0/3542072778 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:27:08.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:08.270+0000 7f9303fff640 1 -- 172.21.15.142:0/3542072778 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f930410ce70 msgr2=0x7f930419e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:27:08.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:08.270+0000 7f9308bb1640 1 --2- 172.21.15.142:0/3542072778 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f930410ce70 0x7f930419e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:27:08.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:08.270+0000 7f9303fff640 1 --2- 172.21.15.142:0/3542072778 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f930410ce70 0x7f930419e010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:08.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:08.270+0000 7f9303fff640 1 -- 172.21.15.142:0/3542072778 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9304104330 msgr2=0x7f9304198c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:27:08.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:08.270+0000 7f9303fff640 1 --2- 172.21.15.142:0/3542072778 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9304104330 0x7f9304198c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:08.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:08.270+0000 7f9303fff640 1 -- 172.21.15.142:0/3542072778 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f930419e7e0 con 0x7f9304103a10 2024-09-06T14:27:08.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:08.270+0000 7f93037fe640 1 --2- 172.21.15.142:0/3542072778 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9304104330 0x7f9304198c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-06T14:27:08.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:08.270+0000 7f9303fff640 1 --2- 172.21.15.142:0/3542072778 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9304103a10 0x7f9304198730 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f92f402a930 tx=0x7f92f40329c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:27:08.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:08.270+0000 7f93017fa640 1 -- 172.21.15.142:0/3542072778 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92f403f030 con 0x7f9304103a10 2024-09-06T14:27:08.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:08.270+0000 7f930a614640 1 -- 172.21.15.142:0/3542072778 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f930419ea70 con 0x7f9304103a10 2024-09-06T14:27:08.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:08.270+0000 7f93017fa640 1 -- 172.21.15.142:0/3542072778 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7f92f40044c0 con 0x7f9304103a10 2024-09-06T14:27:08.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:08.271+0000 7f93017fa640 1 -- 172.21.15.142:0/3542072778 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92f4035070 con 0x7f9304103a10 2024-09-06T14:27:08.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:08.271+0000 7f930a614640 1 -- 172.21.15.142:0/3542072778 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f93041a5190 con 0x7f9304103a10 2024-09-06T14:27:08.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:08.271+0000 7f930a614640 1 -- 172.21.15.142:0/3542072778 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9304103df0 con 0x7f9304103a10 2024-09-06T14:27:08.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:08.272+0000 7f93017fa640 1 -- 172.21.15.142:0/3542072778 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 22) ==== 99844+0+0 (secure 0 0 0) 0x7f92f4042030 con 0x7f9304103a10 2024-09-06T14:27:08.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:08.272+0000 7f93017fa640 1 --2- 172.21.15.142:0/3542072778 >> v2:172.21.15.142:6800/2944545612 conn(0x7f92ec077620 0x7f92ec079ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:27:08.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:08.273+0000 7f93017fa640 1 -- 172.21.15.142:0/3542072778 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(64..64 src has 1..64) ==== 6130+0+0 (secure 0 0 0) 0x7f92f40b9de0 con 0x7f9304103a10 2024-09-06T14:27:08.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:08.273+0000 7f93037fe640 1 --2- 172.21.15.142:0/3542072778 >> v2:172.21.15.142:6800/2944545612 conn(0x7f92ec077620 0x7f92ec079ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:27:08.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:08.273+0000 7f93037fe640 1 --2- 172.21.15.142:0/3542072778 >> v2:172.21.15.142:6800/2944545612 conn(0x7f92ec077620 0x7f92ec079ae0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f9304100bd0 tx=0x7f92f8005f90 comp rx=0 tx=0).ready entity=mgr.14601 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:27:08.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:08.276+0000 7f93017fa640 1 -- 172.21.15.142:0/3542072778 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7f92f4087010 con 0x7f9304103a10 2024-09-06T14:27:08.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:08.487+0000 7f930a614640 1 -- 172.21.15.142:0/3542072778 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f93041075f0 con 0x7f9304103a10 2024-09-06T14:27:08.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:08.487+0000 7f93017fa640 1 -- 172.21.15.142:0/3542072778 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+212 (secure 0 0 0) 0x7f92f408ba30 con 0x7f9304103a10 2024-09-06T14:27:08.485 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:27:08.485 INFO:teuthology.orchestra.run.smithi142.stdout:{"status":"HEALTH_WARN","checks":{"MGR_MODULE_DEPENDENCY":{"severity":"HEALTH_WARN","summary":{"message":"Module 'volumes' has failed dependency: No module named 'ceph.fs'","count":1},"muted":false}},"mutes":[]} 2024-09-06T14:27:08.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:08.489+0000 7f930a614640 1 -- 172.21.15.142:0/3542072778 >> v2:172.21.15.142:6800/2944545612 conn(0x7f92ec077620 msgr2=0x7f92ec079ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:27:08.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:08.489+0000 7f930a614640 1 --2- 172.21.15.142:0/3542072778 >> v2:172.21.15.142:6800/2944545612 conn(0x7f92ec077620 0x7f92ec079ae0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f9304100bd0 tx=0x7f92f8005f90 comp rx=0 tx=0).stop 2024-09-06T14:27:08.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:08.489+0000 7f930a614640 1 -- 172.21.15.142:0/3542072778 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9304103a10 msgr2=0x7f9304198730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:27:08.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:08.489+0000 7f930a614640 1 --2- 172.21.15.142:0/3542072778 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9304103a10 0x7f9304198730 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f92f402a930 tx=0x7f92f40329c0 comp rx=0 tx=0).stop 2024-09-06T14:27:08.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:08.490+0000 7f930a614640 1 -- 172.21.15.142:0/3542072778 shutdown_connections 2024-09-06T14:27:08.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:08.490+0000 7f930a614640 1 --2- 172.21.15.142:0/3542072778 >> v2:172.21.15.142:6800/2944545612 conn(0x7f92ec077620 0x7f92ec079ae0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:08.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:08.490+0000 7f930a614640 1 --2- 172.21.15.142:0/3542072778 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f930410ce70 0x7f930419e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:08.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:08.490+0000 7f930a614640 1 --2- 172.21.15.142:0/3542072778 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9304104330 0x7f9304198c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:08.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:08.490+0000 7f930a614640 1 --2- 172.21.15.142:0/3542072778 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9304103a10 0x7f9304198730 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:08.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:08.490+0000 7f930a614640 1 -- 172.21.15.142:0/3542072778 >> 172.21.15.142:0/3542072778 conn(0x7f93040fd470 msgr2=0x7f93041047c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:27:08.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:08.490+0000 7f930a614640 1 -- 172.21.15.142:0/3542072778 shutdown_connections 2024-09-06T14:27:08.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:08.490+0000 7f930a614640 1 -- 172.21.15.142:0/3542072778 wait complete. 2024-09-06T14:27:09.111 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:27:08 smithi162 ceph-mon[33787]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-06T14:27:09.111 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:27:08 smithi162 ceph-mon[33787]: pgmap v71: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:27:09.112 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:27:08 smithi162 ceph-mon[33787]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-06T14:27:09.112 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:27:08 smithi162 ceph-mon[33787]: from='client.? 172.21.15.142:0/3542072778' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:27:09.192 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:27:08 smithi142 ceph-mon[29958]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-06T14:27:09.192 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:27:08 smithi142 ceph-mon[29958]: pgmap v71: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:27:09.192 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:27:08 smithi142 ceph-mon[29958]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-06T14:27:09.193 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:27:08 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/3542072778' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:27:09.193 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:27:08 smithi142 ceph-mon[37372]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-06T14:27:09.193 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:27:08 smithi142 ceph-mon[37372]: pgmap v71: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:27:09.193 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:27:08 smithi142 ceph-mon[37372]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-06T14:27:09.193 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:27:08 smithi142 ceph-mon[37372]: from='client.? 172.21.15.142:0/3542072778' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:27:10.297 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:27:09 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=infra.usagestats t=2024-09-06T14:27:09.856155426Z level=info msg="Usage stats are ready to report" 2024-09-06T14:27:11.190 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:27:10 smithi162 ceph-mon[33787]: pgmap v72: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:27:11.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:27:10 smithi142 ceph-mon[29958]: pgmap v72: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:27:11.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:27:10 smithi142 ceph-mon[37372]: pgmap v72: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:27:12.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:27:11 smithi142 ceph-mon[29958]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-06T14:27:12.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:27:11 smithi142 ceph-mon[29958]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:27:12.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:27:11 smithi142 ceph-mon[29958]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:27:12.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:27:11 smithi142 ceph-mon[37372]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-06T14:27:12.210 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:27:11 smithi142 ceph-mon[37372]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:27:12.210 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:27:11 smithi142 ceph-mon[37372]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:27:12.223 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph health --format=json 2024-09-06T14:27:12.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:27:11 smithi162 ceph-mon[33787]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-06T14:27:12.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:27:11 smithi162 ceph-mon[33787]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:27:12.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:27:11 smithi162 ceph-mon[33787]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:27:12.472 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:27:12.927 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:27:12 smithi142 ceph-mon[29958]: pgmap v73: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:27:12.927 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:27:12 smithi142 ceph-mon[29958]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:27:12.928 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:27:12 smithi142 ceph-mon[29958]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:27:12.928 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:27:12 smithi142 ceph-mon[29958]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:27:12.928 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:27:12 smithi142 ceph-mon[29958]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-06T14:27:12.928 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:27:12 smithi142 ceph-mon[29958]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:27:12.928 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:27:12 smithi142 ceph-mon[37372]: pgmap v73: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:27:12.928 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:27:12 smithi142 ceph-mon[37372]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:27:12.928 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:27:12 smithi142 ceph-mon[37372]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:27:12.928 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:27:12 smithi142 ceph-mon[37372]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:27:12.928 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:27:12 smithi142 ceph-mon[37372]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-06T14:27:12.928 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:27:12 smithi142 ceph-mon[37372]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:27:13.017 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:13.009+0000 7f9d60105640 1 -- 172.21.15.142:0/2860522756 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9d58104290 msgr2=0x7f9d5810c870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:27:13.017 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:13.009+0000 7f9d60105640 1 --2- 172.21.15.142:0/2860522756 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9d58104290 0x7f9d5810c870 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f9d48009900 tx=0x7f9d4802a410 comp rx=0 tx=0).stop 2024-09-06T14:27:13.017 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:13.010+0000 7f9d60105640 1 -- 172.21.15.142:0/2860522756 shutdown_connections 2024-09-06T14:27:13.017 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:13.010+0000 7f9d60105640 1 --2- 172.21.15.142:0/2860522756 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9d5810ce40 0x7f9d5810f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:13.017 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:13.010+0000 7f9d60105640 1 --2- 172.21.15.142:0/2860522756 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9d58104290 0x7f9d5810c870 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:13.017 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:13.010+0000 7f9d60105640 1 --2- 172.21.15.142:0/2860522756 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9d58103890 0x7f9d58103c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:13.017 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:13.010+0000 7f9d60105640 1 -- 172.21.15.142:0/2860522756 >> 172.21.15.142:0/2860522756 conn(0x7f9d580fd440 msgr2=0x7f9d580ff860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:27:13.017 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:13.010+0000 7f9d60105640 1 -- 172.21.15.142:0/2860522756 shutdown_connections 2024-09-06T14:27:13.017 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:13.010+0000 7f9d60105640 1 -- 172.21.15.142:0/2860522756 wait complete. 2024-09-06T14:27:13.018 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:13.010+0000 7f9d60105640 1 Processor -- start 2024-09-06T14:27:13.018 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:13.010+0000 7f9d60105640 1 -- start start 2024-09-06T14:27:13.018 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:13.010+0000 7f9d60105640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9d58103890 0x7f9d581986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:27:13.018 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:13.010+0000 7f9d60105640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9d58104290 0x7f9d58198be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:27:13.018 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:13.011+0000 7f9d60105640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9d5810ce40 0x7f9d5819df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:27:13.018 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:13.011+0000 7f9d60105640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f9d581121b0 con 0x7f9d58104290 2024-09-06T14:27:13.018 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:13.011+0000 7f9d60105640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f9d58112030 con 0x7f9d5810ce40 2024-09-06T14:27:13.018 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:13.011+0000 7f9d60105640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7f9d58112330 con 0x7f9d58103890 2024-09-06T14:27:13.018 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:13.011+0000 7f9d5dea1640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9d58103890 0x7f9d581986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:27:13.018 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:13.011+0000 7f9d5d6a0640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9d58104290 0x7f9d58198be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:27:13.018 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:13.011+0000 7f9d5d6a0640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9d58104290 0x7f9d58198be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:39302/0 (socket says 172.21.15.142:39302) 2024-09-06T14:27:13.018 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:13.011+0000 7f9d5d6a0640 1 -- 172.21.15.142:0/1598777547 learned_addr learned my addr 172.21.15.142:0/1598777547 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:27:13.018 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:13.011+0000 7f9d5e6a2640 1 --2- 172.21.15.142:0/1598777547 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9d5810ce40 0x7f9d5819df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:27:13.018 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:13.011+0000 7f9d5d6a0640 1 -- 172.21.15.142:0/1598777547 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9d58103890 msgr2=0x7f9d581986a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:27:13.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:13.011+0000 7f9d5d6a0640 1 --2- 172.21.15.142:0/1598777547 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9d58103890 0x7f9d581986a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:13.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:13.011+0000 7f9d5d6a0640 1 -- 172.21.15.142:0/1598777547 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9d5810ce40 msgr2=0x7f9d5819df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:27:13.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:13.011+0000 7f9d5d6a0640 1 --2- 172.21.15.142:0/1598777547 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9d5810ce40 0x7f9d5819df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:13.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:13.011+0000 7f9d5d6a0640 1 -- 172.21.15.142:0/1598777547 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d5819e750 con 0x7f9d58104290 2024-09-06T14:27:13.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:13.011+0000 7f9d5dea1640 1 --2- 172.21.15.142:0/1598777547 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9d58103890 0x7f9d581986a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-06T14:27:13.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:13.011+0000 7f9d5d6a0640 1 --2- 172.21.15.142:0/1598777547 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9d58104290 0x7f9d58198be0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f9d48005940 tx=0x7f9d48032a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:27:13.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:13.011+0000 7f9d4effd640 1 -- 172.21.15.142:0/1598777547 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d4803f030 con 0x7f9d58104290 2024-09-06T14:27:13.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:13.011+0000 7f9d4effd640 1 -- 172.21.15.142:0/1598777547 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7f9d48032e40 con 0x7f9d58104290 2024-09-06T14:27:13.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:13.011+0000 7f9d60105640 1 -- 172.21.15.142:0/1598777547 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d5819e9e0 con 0x7f9d58104290 2024-09-06T14:27:13.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:13.011+0000 7f9d4effd640 1 -- 172.21.15.142:0/1598777547 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d48035070 con 0x7f9d58104290 2024-09-06T14:27:13.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:13.011+0000 7f9d60105640 1 -- 172.21.15.142:0/1598777547 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d581a5100 con 0x7f9d58104290 2024-09-06T14:27:13.020 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:13.012+0000 7f9d60105640 1 -- 172.21.15.142:0/1598777547 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d58103c70 con 0x7f9d58104290 2024-09-06T14:27:13.020 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:13.013+0000 7f9d4effd640 1 -- 172.21.15.142:0/1598777547 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 22) ==== 99844+0+0 (secure 0 0 0) 0x7f9d48042030 con 0x7f9d58104290 2024-09-06T14:27:13.020 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:13.014+0000 7f9d4effd640 1 --2- 172.21.15.142:0/1598777547 >> v2:172.21.15.142:6800/2944545612 conn(0x7f9d44077620 0x7f9d44079ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:27:13.020 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:13.014+0000 7f9d5dea1640 1 --2- 172.21.15.142:0/1598777547 >> v2:172.21.15.142:6800/2944545612 conn(0x7f9d44077620 0x7f9d44079ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:27:13.020 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:13.014+0000 7f9d4effd640 1 -- 172.21.15.142:0/1598777547 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 6130+0+0 (secure 0 0 0) 0x7f9d480b9c70 con 0x7f9d58104290 2024-09-06T14:27:13.020 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:13.014+0000 7f9d5dea1640 1 --2- 172.21.15.142:0/1598777547 >> v2:172.21.15.142:6800/2944545612 conn(0x7f9d44077620 0x7f9d44079ae0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9d540095f0 tx=0x7f9d54005f90 comp rx=0 tx=0).ready entity=mgr.14601 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:27:13.020 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:13.018+0000 7f9d4effd640 1 -- 172.21.15.142:0/1598777547 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7f9d48086f20 con 0x7f9d58104290 2024-09-06T14:27:13.204 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:13.207+0000 7f9d60105640 1 -- 172.21.15.142:0/1598777547 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f9d58107eb0 con 0x7f9d58104290 2024-09-06T14:27:13.204 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:13.207+0000 7f9d4effd640 1 -- 172.21.15.142:0/1598777547 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+212 (secure 0 0 0) 0x7f9d4808b940 con 0x7f9d58104290 2024-09-06T14:27:13.205 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:27:13.205 INFO:teuthology.orchestra.run.smithi142.stdout:{"status":"HEALTH_WARN","checks":{"MGR_MODULE_DEPENDENCY":{"severity":"HEALTH_WARN","summary":{"message":"Module 'volumes' has failed dependency: No module named 'ceph.fs'","count":1},"muted":false}},"mutes":[]} 2024-09-06T14:27:13.207 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:13.209+0000 7f9d60105640 1 -- 172.21.15.142:0/1598777547 >> v2:172.21.15.142:6800/2944545612 conn(0x7f9d44077620 msgr2=0x7f9d44079ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:27:13.207 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:13.209+0000 7f9d60105640 1 --2- 172.21.15.142:0/1598777547 >> v2:172.21.15.142:6800/2944545612 conn(0x7f9d44077620 0x7f9d44079ae0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9d540095f0 tx=0x7f9d54005f90 comp rx=0 tx=0).stop 2024-09-06T14:27:13.207 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:13.209+0000 7f9d60105640 1 -- 172.21.15.142:0/1598777547 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9d58104290 msgr2=0x7f9d58198be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:27:13.207 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:13.209+0000 7f9d60105640 1 --2- 172.21.15.142:0/1598777547 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9d58104290 0x7f9d58198be0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f9d48005940 tx=0x7f9d48032a40 comp rx=0 tx=0).stop 2024-09-06T14:27:13.207 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:13.209+0000 7f9d60105640 1 -- 172.21.15.142:0/1598777547 shutdown_connections 2024-09-06T14:27:13.207 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:13.209+0000 7f9d60105640 1 --2- 172.21.15.142:0/1598777547 >> v2:172.21.15.142:6800/2944545612 conn(0x7f9d44077620 0x7f9d44079ae0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:13.207 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:13.209+0000 7f9d60105640 1 --2- 172.21.15.142:0/1598777547 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9d5810ce40 0x7f9d5819df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:13.207 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:13.209+0000 7f9d60105640 1 --2- 172.21.15.142:0/1598777547 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9d58104290 0x7f9d58198be0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:13.207 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:13.209+0000 7f9d60105640 1 --2- 172.21.15.142:0/1598777547 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9d58103890 0x7f9d581986a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:13.207 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:13.209+0000 7f9d60105640 1 -- 172.21.15.142:0/1598777547 >> 172.21.15.142:0/1598777547 conn(0x7f9d580fd440 msgr2=0x7f9d580fe0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:27:13.208 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:13.210+0000 7f9d60105640 1 -- 172.21.15.142:0/1598777547 shutdown_connections 2024-09-06T14:27:13.208 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:13.210+0000 7f9d60105640 1 -- 172.21.15.142:0/1598777547 wait complete. 2024-09-06T14:27:13.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:27:12 smithi162 ceph-mon[33787]: pgmap v73: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:27:13.298 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:27:12 smithi162 ceph-mon[33787]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:27:13.298 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:27:12 smithi162 ceph-mon[33787]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:27:13.298 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:27:12 smithi162 ceph-mon[33787]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:27:13.298 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:27:12 smithi162 ceph-mon[33787]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-06T14:27:13.298 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:27:12 smithi162 ceph-mon[33787]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:27:14.199 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:27:13 smithi162 ceph-mon[33787]: from='client.? 172.21.15.142:0/1598777547' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:27:14.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:27:13 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/1598777547' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:27:14.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:27:13 smithi142 ceph-mon[37372]: from='client.? 172.21.15.142:0/1598777547' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:27:14.709 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:27:14 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: ::ffff:172.21.15.162 - - [06/Sep/2024:14:27:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-06T14:27:15.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:27:14 smithi142 ceph-mon[29958]: pgmap v74: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:27:15.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:27:14 smithi142 ceph-mon[37372]: pgmap v74: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:27:15.213 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:27:14 smithi162 ceph-mon[33787]: pgmap v74: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:27:17.036 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph health --format=json 2024-09-06T14:27:17.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:27:16 smithi142 ceph-mon[29958]: pgmap v75: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:27:17.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:27:16 smithi142 ceph-mon[37372]: pgmap v75: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:27:17.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:27:16 smithi162 ceph-mon[33787]: pgmap v75: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:27:17.298 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 06 14:27:17 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-iscsi-iscsi-a[59853]: debug there is no tcmu-runner data available 2024-09-06T14:27:17.299 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:27:17.794 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:17.767+0000 7f419bf3f640 1 -- 172.21.15.142:0/3905103786 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4194107d50 msgr2=0x7f419410e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:27:17.794 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:17.767+0000 7f419bf3f640 1 --2- 172.21.15.142:0/3905103786 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4194107d50 0x7f419410e550 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f418c009ca0 tx=0x7f418c02a8c0 comp rx=0 tx=0).stop 2024-09-06T14:27:17.794 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:17.767+0000 7f419bf3f640 1 -- 172.21.15.142:0/3905103786 shutdown_connections 2024-09-06T14:27:17.795 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:17.767+0000 7f419bf3f640 1 --2- 172.21.15.142:0/3905103786 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4194107d50 0x7f419410e550 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:17.795 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:17.767+0000 7f419bf3f640 1 --2- 172.21.15.142:0/3905103786 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f41941073d0 0x7f4194107810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:17.795 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:17.767+0000 7f419bf3f640 1 --2- 172.21.15.142:0/3905103786 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f41941019c0 0x7f4194101da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:17.795 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:17.767+0000 7f419bf3f640 1 -- 172.21.15.142:0/3905103786 >> 172.21.15.142:0/3905103786 conn(0x7f41940fd430 msgr2=0x7f41940ff850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:27:17.795 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:17.767+0000 7f419bf3f640 1 -- 172.21.15.142:0/3905103786 shutdown_connections 2024-09-06T14:27:17.795 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:17.767+0000 7f419bf3f640 1 -- 172.21.15.142:0/3905103786 wait complete. 2024-09-06T14:27:17.795 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:17.767+0000 7f419bf3f640 1 Processor -- start 2024-09-06T14:27:17.795 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:17.767+0000 7f419bf3f640 1 -- start start 2024-09-06T14:27:17.795 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:17.767+0000 7f419bf3f640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f41941019c0 0x7f419419a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:27:17.795 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:17.767+0000 7f419bf3f640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f41941073d0 0x7f419419ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:27:17.795 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:17.767+0000 7f419bf3f640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4194107d50 0x7f41941a01d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:27:17.795 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:17.767+0000 7f419bf3f640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f4194111120 con 0x7f4194107d50 2024-09-06T14:27:17.795 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:17.767+0000 7f419bf3f640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f4194110fa0 con 0x7f41941019c0 2024-09-06T14:27:17.795 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:17.767+0000 7f419bf3f640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7f41941112a0 con 0x7f41941073d0 2024-09-06T14:27:17.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:17.768+0000 7f41994da640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f41941073d0 0x7f419419ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:27:17.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:17.768+0000 7f419a4dc640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4194107d50 0x7f41941a01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:27:17.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:17.768+0000 7f41994da640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f41941073d0 0x7f419419ae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:35722/0 (socket says 172.21.15.142:35722) 2024-09-06T14:27:17.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:17.768+0000 7f41994da640 1 -- 172.21.15.142:0/769465351 learned_addr learned my addr 172.21.15.142:0/769465351 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:27:17.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:17.768+0000 7f419a4dc640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4194107d50 0x7f41941a01d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:46052/0 (socket says 172.21.15.142:46052) 2024-09-06T14:27:17.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:17.768+0000 7f4199cdb640 1 --2- 172.21.15.142:0/769465351 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f41941019c0 0x7f419419a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:27:17.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:17.768+0000 7f41994da640 1 -- 172.21.15.142:0/769465351 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f41941019c0 msgr2=0x7f419419a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:27:17.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:17.768+0000 7f41994da640 1 --2- 172.21.15.142:0/769465351 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f41941019c0 0x7f419419a8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:17.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:17.768+0000 7f41994da640 1 -- 172.21.15.142:0/769465351 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4194107d50 msgr2=0x7f41941a01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:27:17.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:17.768+0000 7f41994da640 1 --2- 172.21.15.142:0/769465351 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4194107d50 0x7f41941a01d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:17.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:17.768+0000 7f41994da640 1 -- 172.21.15.142:0/769465351 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41941a09a0 con 0x7f41941073d0 2024-09-06T14:27:17.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:17.768+0000 7f419a4dc640 1 --2- 172.21.15.142:0/769465351 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4194107d50 0x7f41941a01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-06T14:27:17.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:17.768+0000 7f4199cdb640 1 --2- 172.21.15.142:0/769465351 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f41941019c0 0x7f419419a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-06T14:27:17.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:17.768+0000 7f41994da640 1 --2- 172.21.15.142:0/769465351 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f41941073d0 0x7f419419ae30 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f418800ea60 tx=0x7f418800ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:27:17.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:17.768+0000 7f4186ffd640 1 -- 172.21.15.142:0/769465351 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f418800cde0 con 0x7f41941073d0 2024-09-06T14:27:17.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:17.768+0000 7f419bf3f640 1 -- 172.21.15.142:0/769465351 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41941a0c90 con 0x7f41941073d0 2024-09-06T14:27:17.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:17.768+0000 7f4186ffd640 1 -- 172.21.15.142:0/769465351 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7f41880035f0 con 0x7f41941073d0 2024-09-06T14:27:17.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:17.768+0000 7f4186ffd640 1 -- 172.21.15.142:0/769465351 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41880044a0 con 0x7f41941073d0 2024-09-06T14:27:17.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:17.768+0000 7f419bf3f640 1 -- 172.21.15.142:0/769465351 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f41941a73d0 con 0x7f41941073d0 2024-09-06T14:27:17.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:17.769+0000 7f419bf3f640 1 -- 172.21.15.142:0/769465351 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4194101da0 con 0x7f41941073d0 2024-09-06T14:27:17.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:17.770+0000 7f4186ffd640 1 -- 172.21.15.142:0/769465351 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 22) ==== 99844+0+0 (secure 0 0 0) 0x7f4188004000 con 0x7f41941073d0 2024-09-06T14:27:17.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:17.770+0000 7f4186ffd640 1 --2- 172.21.15.142:0/769465351 >> v2:172.21.15.142:6800/2944545612 conn(0x7f4180077620 0x7f4180079ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:27:17.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:17.770+0000 7f4186ffd640 1 -- 172.21.15.142:0/769465351 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(64..64 src has 1..64) ==== 6130+0+0 (secure 0 0 0) 0x7f4188099d80 con 0x7f41941073d0 2024-09-06T14:27:17.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:17.770+0000 7f4199cdb640 1 --2- 172.21.15.142:0/769465351 >> v2:172.21.15.142:6800/2944545612 conn(0x7f4180077620 0x7f4180079ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:27:17.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:17.771+0000 7f4199cdb640 1 --2- 172.21.15.142:0/769465351 >> v2:172.21.15.142:6800/2944545612 conn(0x7f4180077620 0x7f4180079ae0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4190006fd0 tx=0x7f4190008040 comp rx=0 tx=0).ready entity=mgr.14601 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:27:17.798 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:17.774+0000 7f4186ffd640 1 -- 172.21.15.142:0/769465351 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7f4188062e40 con 0x7f41941073d0 2024-09-06T14:27:17.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:17.963+0000 7f419bf3f640 1 -- 172.21.15.142:0/769465351 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f41941a7690 con 0x7f41941073d0 2024-09-06T14:27:17.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:17.964+0000 7f4186ffd640 1 -- 172.21.15.142:0/769465351 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+212 (secure 0 0 0) 0x7f4188083e20 con 0x7f41941073d0 2024-09-06T14:27:17.961 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:27:17.961 INFO:teuthology.orchestra.run.smithi142.stdout:{"status":"HEALTH_WARN","checks":{"MGR_MODULE_DEPENDENCY":{"severity":"HEALTH_WARN","summary":{"message":"Module 'volumes' has failed dependency: No module named 'ceph.fs'","count":1},"muted":false}},"mutes":[]} 2024-09-06T14:27:17.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:17.966+0000 7f419bf3f640 1 -- 172.21.15.142:0/769465351 >> v2:172.21.15.142:6800/2944545612 conn(0x7f4180077620 msgr2=0x7f4180079ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:27:17.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:17.966+0000 7f419bf3f640 1 --2- 172.21.15.142:0/769465351 >> v2:172.21.15.142:6800/2944545612 conn(0x7f4180077620 0x7f4180079ae0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4190006fd0 tx=0x7f4190008040 comp rx=0 tx=0).stop 2024-09-06T14:27:17.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:17.966+0000 7f419bf3f640 1 -- 172.21.15.142:0/769465351 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f41941073d0 msgr2=0x7f419419ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:27:17.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:17.966+0000 7f419bf3f640 1 --2- 172.21.15.142:0/769465351 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f41941073d0 0x7f419419ae30 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f418800ea60 tx=0x7f418800ef20 comp rx=0 tx=0).stop 2024-09-06T14:27:17.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:17.966+0000 7f419bf3f640 1 -- 172.21.15.142:0/769465351 shutdown_connections 2024-09-06T14:27:17.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:17.966+0000 7f419bf3f640 1 --2- 172.21.15.142:0/769465351 >> v2:172.21.15.142:6800/2944545612 conn(0x7f4180077620 0x7f4180079ae0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:17.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:17.966+0000 7f419bf3f640 1 --2- 172.21.15.142:0/769465351 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4194107d50 0x7f41941a01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:17.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:17.966+0000 7f419bf3f640 1 --2- 172.21.15.142:0/769465351 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f41941073d0 0x7f419419ae30 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:17.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:17.966+0000 7f419bf3f640 1 --2- 172.21.15.142:0/769465351 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f41941019c0 0x7f419419a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:17.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:17.966+0000 7f419bf3f640 1 -- 172.21.15.142:0/769465351 >> 172.21.15.142:0/769465351 conn(0x7f41940fd430 msgr2=0x7f419406b000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:27:17.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:17.966+0000 7f419bf3f640 1 -- 172.21.15.142:0/769465351 shutdown_connections 2024-09-06T14:27:17.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:17.966+0000 7f419bf3f640 1 -- 172.21.15.142:0/769465351 wait complete. 2024-09-06T14:27:19.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:27:18 smithi142 ceph-mon[29958]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-06T14:27:19.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:27:18 smithi142 ceph-mon[29958]: pgmap v76: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:27:19.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:27:18 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/769465351' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:27:19.210 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:27:18 smithi142 ceph-mon[37372]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-06T14:27:19.210 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:27:18 smithi142 ceph-mon[37372]: pgmap v76: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:27:19.210 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:27:18 smithi142 ceph-mon[37372]: from='client.? 172.21.15.142:0/769465351' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:27:19.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:27:18 smithi162 ceph-mon[33787]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-06T14:27:19.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:27:18 smithi162 ceph-mon[33787]: pgmap v76: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:27:19.298 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:27:18 smithi162 ceph-mon[33787]: from='client.? 172.21.15.142:0/769465351' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:27:21.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:27:20 smithi142 ceph-mon[37372]: pgmap v77: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:27:21.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:27:20 smithi142 ceph-mon[29958]: pgmap v77: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:27:21.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:27:20 smithi162 ceph-mon[33787]: pgmap v77: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:27:21.778 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph health --format=json 2024-09-06T14:27:22.042 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:27:22.544 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:22.518+0000 7f526cae9640 1 -- 172.21.15.142:0/3967936196 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5268107d50 msgr2=0x7f526810e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:27:22.544 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:22.518+0000 7f526cae9640 1 --2- 172.21.15.142:0/3967936196 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5268107d50 0x7f526810e550 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f5260009860 tx=0x7f526002a340 comp rx=0 tx=0).stop 2024-09-06T14:27:22.544 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:22.518+0000 7f526cae9640 1 -- 172.21.15.142:0/3967936196 shutdown_connections 2024-09-06T14:27:22.544 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:22.518+0000 7f526cae9640 1 --2- 172.21.15.142:0/3967936196 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5268107d50 0x7f526810e550 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:22.544 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:22.518+0000 7f526cae9640 1 --2- 172.21.15.142:0/3967936196 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f52681073d0 0x7f5268107810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:22.544 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:22.518+0000 7f526cae9640 1 --2- 172.21.15.142:0/3967936196 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f52681019c0 0x7f5268101da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:22.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:22.518+0000 7f526cae9640 1 -- 172.21.15.142:0/3967936196 >> 172.21.15.142:0/3967936196 conn(0x7f52680fd430 msgr2=0x7f52680ff850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:27:22.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:22.518+0000 7f526cae9640 1 -- 172.21.15.142:0/3967936196 shutdown_connections 2024-09-06T14:27:22.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:22.518+0000 7f526cae9640 1 -- 172.21.15.142:0/3967936196 wait complete. 2024-09-06T14:27:22.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:22.519+0000 7f526cae9640 1 Processor -- start 2024-09-06T14:27:22.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:22.519+0000 7f526cae9640 1 -- start start 2024-09-06T14:27:22.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:22.519+0000 7f526cae9640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f52681019c0 0x7f5268197630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:27:22.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:22.519+0000 7f526cae9640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f52681073d0 0x7f5268197b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:27:22.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:22.519+0000 7f526cae9640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5268107d50 0x7f526819cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:27:22.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:22.519+0000 7f526cae9640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f5268111120 con 0x7f52681019c0 2024-09-06T14:27:22.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:22.519+0000 7f526cae9640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5268110fa0 con 0x7f5268107d50 2024-09-06T14:27:22.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:22.519+0000 7f526cae9640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7f52681112a0 con 0x7f52681073d0 2024-09-06T14:27:22.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:22.519+0000 7f526659c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f52681019c0 0x7f5268197630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:27:22.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:22.519+0000 7f5265d9b640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f52681073d0 0x7f5268197b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:27:22.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:22.519+0000 7f526659c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f52681019c0 0x7f5268197630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:46080/0 (socket says 172.21.15.142:46080) 2024-09-06T14:27:22.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:22.519+0000 7f526659c640 1 -- 172.21.15.142:0/1633484837 learned_addr learned my addr 172.21.15.142:0/1633484837 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:27:22.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:22.519+0000 7f526659c640 1 -- 172.21.15.142:0/1633484837 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f52681073d0 msgr2=0x7f5268197b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:27:22.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:22.519+0000 7f526659c640 1 --2- 172.21.15.142:0/1633484837 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f52681073d0 0x7f5268197b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:22.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:22.519+0000 7f5266d9d640 1 --2- 172.21.15.142:0/1633484837 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5268107d50 0x7f526819cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:27:22.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:22.519+0000 7f526659c640 1 -- 172.21.15.142:0/1633484837 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5268107d50 msgr2=0x7f526819cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:27:22.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:22.519+0000 7f526659c640 1 --2- 172.21.15.142:0/1633484837 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5268107d50 0x7f526819cf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:22.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:22.519+0000 7f526659c640 1 -- 172.21.15.142:0/1633484837 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f526819d6e0 con 0x7f52681019c0 2024-09-06T14:27:22.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:22.519+0000 7f5266d9d640 1 --2- 172.21.15.142:0/1633484837 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5268107d50 0x7f526819cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-06T14:27:22.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:22.520+0000 7f526659c640 1 --2- 172.21.15.142:0/1633484837 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f52681019c0 0x7f5268197630 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f525800df40 tx=0x7f525800df70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:27:22.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:22.520+0000 7f525f7fe640 1 -- 172.21.15.142:0/1633484837 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f525800ea60 con 0x7f52681019c0 2024-09-06T14:27:22.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:22.520+0000 7f525f7fe640 1 -- 172.21.15.142:0/1633484837 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7f525800ec00 con 0x7f52681019c0 2024-09-06T14:27:22.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:22.520+0000 7f526cae9640 1 -- 172.21.15.142:0/1633484837 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f526819d970 con 0x7f52681019c0 2024-09-06T14:27:22.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:22.520+0000 7f525f7fe640 1 -- 172.21.15.142:0/1633484837 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5258016630 con 0x7f52681019c0 2024-09-06T14:27:22.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:22.520+0000 7f526cae9640 1 -- 172.21.15.142:0/1633484837 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f526806a860 con 0x7f52681019c0 2024-09-06T14:27:22.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:22.521+0000 7f526cae9640 1 -- 172.21.15.142:0/1633484837 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5268101da0 con 0x7f52681019c0 2024-09-06T14:27:22.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:22.521+0000 7f525f7fe640 1 -- 172.21.15.142:0/1633484837 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 22) ==== 99844+0+0 (secure 0 0 0) 0x7f5258009580 con 0x7f52681019c0 2024-09-06T14:27:22.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:22.522+0000 7f525f7fe640 1 --2- 172.21.15.142:0/1633484837 >> v2:172.21.15.142:6800/2944545612 conn(0x7f5254077620 0x7f5254079ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:27:22.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:22.522+0000 7f5265d9b640 1 --2- 172.21.15.142:0/1633484837 >> v2:172.21.15.142:6800/2944545612 conn(0x7f5254077620 0x7f5254079ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:27:22.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:22.522+0000 7f525f7fe640 1 -- 172.21.15.142:0/1633484837 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 6130+0+0 (secure 0 0 0) 0x7f525809afe0 con 0x7f52681019c0 2024-09-06T14:27:22.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:22.523+0000 7f5265d9b640 1 --2- 172.21.15.142:0/1633484837 >> v2:172.21.15.142:6800/2944545612 conn(0x7f5254077620 0x7f5254079ae0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f5268105ea0 tx=0x7f5250009260 comp rx=0 tx=0).ready entity=mgr.14601 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:27:22.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:22.526+0000 7f525f7fe640 1 -- 172.21.15.142:0/1633484837 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7f52580681b0 con 0x7f52681019c0 2024-09-06T14:27:22.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:22.716+0000 7f526cae9640 1 -- 172.21.15.142:0/1633484837 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f526806b280 con 0x7f52681019c0 2024-09-06T14:27:22.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:22.716+0000 7f525f7fe640 1 -- 172.21.15.142:0/1633484837 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+212 (secure 0 0 0) 0x7f525806cbd0 con 0x7f52681019c0 2024-09-06T14:27:22.714 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:27:22.714 INFO:teuthology.orchestra.run.smithi142.stdout:{"status":"HEALTH_WARN","checks":{"MGR_MODULE_DEPENDENCY":{"severity":"HEALTH_WARN","summary":{"message":"Module 'volumes' has failed dependency: No module named 'ceph.fs'","count":1},"muted":false}},"mutes":[]} 2024-09-06T14:27:22.715 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:22.718+0000 7f526cae9640 1 -- 172.21.15.142:0/1633484837 >> v2:172.21.15.142:6800/2944545612 conn(0x7f5254077620 msgr2=0x7f5254079ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:27:22.716 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:22.718+0000 7f526cae9640 1 --2- 172.21.15.142:0/1633484837 >> v2:172.21.15.142:6800/2944545612 conn(0x7f5254077620 0x7f5254079ae0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f5268105ea0 tx=0x7f5250009260 comp rx=0 tx=0).stop 2024-09-06T14:27:22.716 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:22.718+0000 7f526cae9640 1 -- 172.21.15.142:0/1633484837 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f52681019c0 msgr2=0x7f5268197630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:27:22.716 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:22.718+0000 7f526cae9640 1 --2- 172.21.15.142:0/1633484837 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f52681019c0 0x7f5268197630 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f525800df40 tx=0x7f525800df70 comp rx=0 tx=0).stop 2024-09-06T14:27:22.716 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:22.718+0000 7f526cae9640 1 -- 172.21.15.142:0/1633484837 shutdown_connections 2024-09-06T14:27:22.716 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:22.718+0000 7f526cae9640 1 --2- 172.21.15.142:0/1633484837 >> v2:172.21.15.142:6800/2944545612 conn(0x7f5254077620 0x7f5254079ae0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:22.716 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:22.718+0000 7f526cae9640 1 --2- 172.21.15.142:0/1633484837 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5268107d50 0x7f526819cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:22.716 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:22.718+0000 7f526cae9640 1 --2- 172.21.15.142:0/1633484837 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f52681073d0 0x7f5268197b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:22.716 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:22.718+0000 7f526cae9640 1 --2- 172.21.15.142:0/1633484837 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f52681019c0 0x7f5268197630 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:22.716 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:22.718+0000 7f526cae9640 1 -- 172.21.15.142:0/1633484837 >> 172.21.15.142:0/1633484837 conn(0x7f52680fd430 msgr2=0x7f52680ff820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:27:22.716 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:22.718+0000 7f526cae9640 1 -- 172.21.15.142:0/1633484837 shutdown_connections 2024-09-06T14:27:22.716 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:22.718+0000 7f526cae9640 1 -- 172.21.15.142:0/1633484837 wait complete. 2024-09-06T14:27:23.003 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:27:22 smithi142 ceph-mon[29958]: pgmap v78: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:27:23.004 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:27:22 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/1633484837' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:27:23.004 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:27:22 smithi142 ceph-mon[37372]: pgmap v78: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:27:23.004 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:27:22 smithi142 ceph-mon[37372]: from='client.? 172.21.15.142:0/1633484837' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:27:23.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:27:22 smithi162 ceph-mon[33787]: pgmap v78: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:27:23.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:27:22 smithi162 ceph-mon[33787]: from='client.? 172.21.15.142:0/1633484837' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:27:24.709 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:27:24 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: ::ffff:172.21.15.162 - - [06/Sep/2024:14:27:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-06T14:27:25.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:27:24 smithi142 ceph-mon[29958]: pgmap v79: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:27:25.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:27:24 smithi142 ceph-mon[37372]: pgmap v79: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:27:25.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:27:24 smithi162 ceph-mon[33787]: pgmap v79: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:27:26.455 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph health --format=json 2024-09-06T14:27:26.719 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:27:27.094 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:27:26 smithi142 ceph-mon[29958]: pgmap v80: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:27:27.094 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:27:26 smithi142 ceph-mon[29958]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-06T14:27:27.094 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:27:26 smithi142 ceph-mon[37372]: pgmap v80: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:27:27.094 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:27:26 smithi142 ceph-mon[37372]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-06T14:27:27.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:27.176+0000 7ffb8a5dd640 1 -- 172.21.15.142:0/926696207 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffb84107d50 msgr2=0x7ffb8410e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:27:27.194 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:27.176+0000 7ffb8a5dd640 1 --2- 172.21.15.142:0/926696207 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffb84107d50 0x7ffb8410e550 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ffb7c009ca0 tx=0x7ffb7c02a8c0 comp rx=0 tx=0).stop 2024-09-06T14:27:27.194 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:27.176+0000 7ffb8a5dd640 1 -- 172.21.15.142:0/926696207 shutdown_connections 2024-09-06T14:27:27.194 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:27.176+0000 7ffb8a5dd640 1 --2- 172.21.15.142:0/926696207 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffb84107d50 0x7ffb8410e550 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:27.194 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:27.176+0000 7ffb8a5dd640 1 --2- 172.21.15.142:0/926696207 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffb841073d0 0x7ffb84107810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:27.194 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:27.176+0000 7ffb8a5dd640 1 --2- 172.21.15.142:0/926696207 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffb841019c0 0x7ffb84101da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:27.194 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:27.176+0000 7ffb8a5dd640 1 -- 172.21.15.142:0/926696207 >> 172.21.15.142:0/926696207 conn(0x7ffb840fd430 msgr2=0x7ffb840ff850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:27:27.194 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:27.176+0000 7ffb8a5dd640 1 -- 172.21.15.142:0/926696207 shutdown_connections 2024-09-06T14:27:27.194 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:27.176+0000 7ffb8a5dd640 1 -- 172.21.15.142:0/926696207 wait complete. 2024-09-06T14:27:27.194 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:27.176+0000 7ffb8a5dd640 1 Processor -- start 2024-09-06T14:27:27.194 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:27.177+0000 7ffb8a5dd640 1 -- start start 2024-09-06T14:27:27.194 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:27.177+0000 7ffb8a5dd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffb841019c0 0x7ffb84199550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:27:27.194 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:27.177+0000 7ffb8a5dd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffb841073d0 0x7ffb84199a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:27:27.194 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:27.177+0000 7ffb8a5dd640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffb84107d50 0x7ffb84199ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:27:27.194 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:27.177+0000 7ffb8a5dd640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7ffb84111170 con 0x7ffb841019c0 2024-09-06T14:27:27.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:27.177+0000 7ffb8a5dd640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ffb84110ff0 con 0x7ffb841073d0 2024-09-06T14:27:27.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:27.177+0000 7ffb8a5dd640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7ffb841112f0 con 0x7ffb84107d50 2024-09-06T14:27:27.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:27.177+0000 7ffb83fff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffb841019c0 0x7ffb84199550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:27:27.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:27.177+0000 7ffb88b7a640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffb84107d50 0x7ffb84199ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:27:27.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:27.177+0000 7ffb83fff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffb841019c0 0x7ffb84199550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:46106/0 (socket says 172.21.15.142:46106) 2024-09-06T14:27:27.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:27.177+0000 7ffb88b7a640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffb84107d50 0x7ffb84199ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:35758/0 (socket says 172.21.15.142:35758) 2024-09-06T14:27:27.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:27.177+0000 7ffb83fff640 1 -- 172.21.15.142:0/4138463606 learned_addr learned my addr 172.21.15.142:0/4138463606 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:27:27.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:27.177+0000 7ffb7bfff640 1 --2- 172.21.15.142:0/4138463606 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffb841073d0 0x7ffb84199a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:27:27.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:27.177+0000 7ffb88b7a640 1 -- 172.21.15.142:0/4138463606 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffb841073d0 msgr2=0x7ffb84199a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:27:27.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:27.177+0000 7ffb88b7a640 1 --2- 172.21.15.142:0/4138463606 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffb841073d0 0x7ffb84199a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:27.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:27.177+0000 7ffb88b7a640 1 -- 172.21.15.142:0/4138463606 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffb841019c0 msgr2=0x7ffb84199550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:27:27.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:27.177+0000 7ffb88b7a640 1 --2- 172.21.15.142:0/4138463606 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffb841019c0 0x7ffb84199550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:27.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:27.177+0000 7ffb88b7a640 1 -- 172.21.15.142:0/4138463606 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb841a4930 con 0x7ffb84107d50 2024-09-06T14:27:27.196 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:27.177+0000 7ffb83fff640 1 --2- 172.21.15.142:0/4138463606 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffb841019c0 0x7ffb84199550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-06T14:27:27.196 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:27.177+0000 7ffb7bfff640 1 --2- 172.21.15.142:0/4138463606 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffb841073d0 0x7ffb84199a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-06T14:27:27.196 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:27.177+0000 7ffb88b7a640 1 --2- 172.21.15.142:0/4138463606 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffb84107d50 0x7ffb84199ff0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7ffb7c005e30 tx=0x7ffb7c0093d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:27:27.196 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:27.177+0000 7ffb81ffb640 1 -- 172.21.15.142:0/4138463606 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb7c04c070 con 0x7ffb84107d50 2024-09-06T14:27:27.196 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:27.178+0000 7ffb81ffb640 1 -- 172.21.15.142:0/4138463606 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7ffb7c003c80 con 0x7ffb84107d50 2024-09-06T14:27:27.196 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:27.178+0000 7ffb8a5dd640 1 -- 172.21.15.142:0/4138463606 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb84197340 con 0x7ffb84107d50 2024-09-06T14:27:27.196 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:27.178+0000 7ffb81ffb640 1 -- 172.21.15.142:0/4138463606 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb7c004520 con 0x7ffb84107d50 2024-09-06T14:27:27.196 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:27.178+0000 7ffb8a5dd640 1 -- 172.21.15.142:0/4138463606 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb84197900 con 0x7ffb84107d50 2024-09-06T14:27:27.196 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:27.179+0000 7ffb8a5dd640 1 -- 172.21.15.142:0/4138463606 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb84101da0 con 0x7ffb84107d50 2024-09-06T14:27:27.196 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:27.179+0000 7ffb81ffb640 1 -- 172.21.15.142:0/4138463606 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 22) ==== 99844+0+0 (secure 0 0 0) 0x7ffb7c04f070 con 0x7ffb84107d50 2024-09-06T14:27:27.196 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:27.180+0000 7ffb81ffb640 1 --2- 172.21.15.142:0/4138463606 >> v2:172.21.15.142:6800/2944545612 conn(0x7ffb70077620 0x7ffb70079ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:27:27.196 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:27.180+0000 7ffb83fff640 1 --2- 172.21.15.142:0/4138463606 >> v2:172.21.15.142:6800/2944545612 conn(0x7ffb70077620 0x7ffb70079ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:27:27.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:27.180+0000 7ffb81ffb640 1 -- 172.21.15.142:0/4138463606 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(64..64 src has 1..64) ==== 6130+0+0 (secure 0 0 0) 0x7ffb7c0bdf30 con 0x7ffb84107d50 2024-09-06T14:27:27.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:27.180+0000 7ffb83fff640 1 --2- 172.21.15.142:0/4138463606 >> v2:172.21.15.142:6800/2944545612 conn(0x7ffb70077620 0x7ffb70079ae0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ffb74009e10 tx=0x7ffb740091c0 comp rx=0 tx=0).ready entity=mgr.14601 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:27:27.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:27.184+0000 7ffb81ffb640 1 -- 172.21.15.142:0/4138463606 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7ffb7c08b160 con 0x7ffb84107d50 2024-09-06T14:27:27.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:27:26 smithi162 ceph-mon[33787]: pgmap v80: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:27:27.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:27:26 smithi162 ceph-mon[33787]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-06T14:27:27.297 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 06 14:27:27 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-iscsi-iscsi-a[59853]: debug there is no tcmu-runner data available 2024-09-06T14:27:27.369 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:27.371+0000 7ffb8a5dd640 1 -- 172.21.15.142:0/4138463606 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7ffb84109b30 con 0x7ffb84107d50 2024-09-06T14:27:27.370 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:27.371+0000 7ffb81ffb640 1 -- 172.21.15.142:0/4138463606 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+212 (secure 0 0 0) 0x7ffb7c08fb80 con 0x7ffb84107d50 2024-09-06T14:27:27.370 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:27:27.370 INFO:teuthology.orchestra.run.smithi142.stdout:{"status":"HEALTH_WARN","checks":{"MGR_MODULE_DEPENDENCY":{"severity":"HEALTH_WARN","summary":{"message":"Module 'volumes' has failed dependency: No module named 'ceph.fs'","count":1},"muted":false}},"mutes":[]} 2024-09-06T14:27:27.372 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:27.374+0000 7ffb8a5dd640 1 -- 172.21.15.142:0/4138463606 >> v2:172.21.15.142:6800/2944545612 conn(0x7ffb70077620 msgr2=0x7ffb70079ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:27:27.372 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:27.374+0000 7ffb8a5dd640 1 --2- 172.21.15.142:0/4138463606 >> v2:172.21.15.142:6800/2944545612 conn(0x7ffb70077620 0x7ffb70079ae0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ffb74009e10 tx=0x7ffb740091c0 comp rx=0 tx=0).stop 2024-09-06T14:27:27.372 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:27.374+0000 7ffb8a5dd640 1 -- 172.21.15.142:0/4138463606 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffb84107d50 msgr2=0x7ffb84199ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:27:27.372 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:27.374+0000 7ffb8a5dd640 1 --2- 172.21.15.142:0/4138463606 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffb84107d50 0x7ffb84199ff0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7ffb7c005e30 tx=0x7ffb7c0093d0 comp rx=0 tx=0).stop 2024-09-06T14:27:27.372 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:27.374+0000 7ffb8a5dd640 1 -- 172.21.15.142:0/4138463606 shutdown_connections 2024-09-06T14:27:27.372 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:27.374+0000 7ffb8a5dd640 1 --2- 172.21.15.142:0/4138463606 >> v2:172.21.15.142:6800/2944545612 conn(0x7ffb70077620 0x7ffb70079ae0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:27.372 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:27.374+0000 7ffb8a5dd640 1 --2- 172.21.15.142:0/4138463606 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffb84107d50 0x7ffb84199ff0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:27.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:27.374+0000 7ffb8a5dd640 1 --2- 172.21.15.142:0/4138463606 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffb841073d0 0x7ffb84199a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:27.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:27.374+0000 7ffb8a5dd640 1 --2- 172.21.15.142:0/4138463606 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffb841019c0 0x7ffb84199550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:27.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:27.374+0000 7ffb8a5dd640 1 -- 172.21.15.142:0/4138463606 >> 172.21.15.142:0/4138463606 conn(0x7ffb840fd430 msgr2=0x7ffb840ff820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:27:27.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:27.374+0000 7ffb8a5dd640 1 -- 172.21.15.142:0/4138463606 shutdown_connections 2024-09-06T14:27:27.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:27.374+0000 7ffb8a5dd640 1 -- 172.21.15.142:0/4138463606 wait complete. 2024-09-06T14:27:28.123 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:27:27 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/4138463606' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:27:28.124 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:27:27 smithi142 ceph-mon[37372]: from='client.? 172.21.15.142:0/4138463606' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:27:28.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:27:27 smithi162 ceph-mon[33787]: from='client.? 172.21.15.142:0/4138463606' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:27:29.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:27:28 smithi142 ceph-mon[37372]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-06T14:27:29.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:27:28 smithi142 ceph-mon[37372]: pgmap v81: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:27:29.210 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:27:28 smithi142 ceph-mon[29958]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-06T14:27:29.210 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:27:28 smithi142 ceph-mon[29958]: pgmap v81: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:27:29.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:27:28 smithi162 ceph-mon[33787]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-06T14:27:29.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:27:28 smithi162 ceph-mon[33787]: pgmap v81: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:27:31.155 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph health --format=json 2024-09-06T14:27:31.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:27:30 smithi142 ceph-mon[29958]: pgmap v82: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:27:31.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:27:30 smithi142 ceph-mon[37372]: pgmap v82: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:27:31.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:27:30 smithi162 ceph-mon[33787]: pgmap v82: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:27:31.415 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:27:31.877 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:31.874+0000 7f8a1ba96640 1 -- 172.21.15.142:0/2756095679 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8a14107d60 msgr2=0x7f8a1410e560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:27:31.877 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:31.874+0000 7f8a1ba96640 1 --2- 172.21.15.142:0/2756095679 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8a14107d60 0x7f8a1410e560 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f8a0c009ca0 tx=0x7f8a0c02a8c0 comp rx=0 tx=0).stop 2024-09-06T14:27:31.877 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:31.874+0000 7f8a1ba96640 1 -- 172.21.15.142:0/2756095679 shutdown_connections 2024-09-06T14:27:31.877 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:31.874+0000 7f8a1ba96640 1 --2- 172.21.15.142:0/2756095679 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8a14107d60 0x7f8a1410e560 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:31.878 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:31.874+0000 7f8a1ba96640 1 --2- 172.21.15.142:0/2756095679 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8a141073e0 0x7f8a14107820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:31.878 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:31.874+0000 7f8a1ba96640 1 --2- 172.21.15.142:0/2756095679 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8a141019d0 0x7f8a14101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:31.878 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:31.874+0000 7f8a1ba96640 1 -- 172.21.15.142:0/2756095679 >> 172.21.15.142:0/2756095679 conn(0x7f8a140fd460 msgr2=0x7f8a140ff880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:27:31.878 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:31.874+0000 7f8a1ba96640 1 -- 172.21.15.142:0/2756095679 shutdown_connections 2024-09-06T14:27:31.878 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:31.874+0000 7f8a1ba96640 1 -- 172.21.15.142:0/2756095679 wait complete. 2024-09-06T14:27:31.878 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:31.875+0000 7f8a1ba96640 1 Processor -- start 2024-09-06T14:27:31.878 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:31.875+0000 7f8a1ba96640 1 -- start start 2024-09-06T14:27:31.878 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:31.875+0000 7f8a1ba96640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8a141019d0 0x7f8a1410f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:27:31.878 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:31.875+0000 7f8a1ba96640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8a141073e0 0x7f8a1410f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:27:31.878 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:31.875+0000 7f8a1ba96640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8a14107d60 0x7f8a1410faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:27:31.878 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:31.875+0000 7f8a1ba96640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f8a14112230 con 0x7f8a141073e0 2024-09-06T14:27:31.878 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:31.875+0000 7f8a1ba96640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f8a141120b0 con 0x7f8a141019d0 2024-09-06T14:27:31.878 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:31.875+0000 7f8a1ba96640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7f8a141123b0 con 0x7f8a14107d60 2024-09-06T14:27:31.879 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:31.875+0000 7f8a19031640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8a141073e0 0x7f8a1410f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:27:31.879 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:31.875+0000 7f8a1a033640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8a14107d60 0x7f8a1410faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:27:31.879 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:31.875+0000 7f8a19031640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8a141073e0 0x7f8a1410f5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:50068/0 (socket says 172.21.15.142:50068) 2024-09-06T14:27:31.879 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:31.875+0000 7f8a19031640 1 -- 172.21.15.142:0/703998198 learned_addr learned my addr 172.21.15.142:0/703998198 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:27:31.879 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:31.875+0000 7f8a1a033640 1 --2- 172.21.15.142:0/703998198 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8a14107d60 0x7f8a1410faf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:55386/0 (socket says 172.21.15.142:55386) 2024-09-06T14:27:31.879 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:31.875+0000 7f8a19832640 1 --2- 172.21.15.142:0/703998198 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8a141019d0 0x7f8a1410f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:27:31.879 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:31.875+0000 7f8a19031640 1 -- 172.21.15.142:0/703998198 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8a14107d60 msgr2=0x7f8a1410faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:27:31.879 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:31.875+0000 7f8a19031640 1 --2- 172.21.15.142:0/703998198 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8a14107d60 0x7f8a1410faf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:31.879 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:31.875+0000 7f8a19031640 1 -- 172.21.15.142:0/703998198 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8a141019d0 msgr2=0x7f8a1410f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:27:31.879 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:31.875+0000 7f8a19031640 1 --2- 172.21.15.142:0/703998198 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8a141019d0 0x7f8a1410f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:31.879 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:31.875+0000 7f8a19031640 1 -- 172.21.15.142:0/703998198 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a141a6a40 con 0x7f8a141073e0 2024-09-06T14:27:31.879 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:31.875+0000 7f8a1a033640 1 --2- 172.21.15.142:0/703998198 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8a14107d60 0x7f8a1410faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-06T14:27:31.879 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:31.875+0000 7f8a19031640 1 --2- 172.21.15.142:0/703998198 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8a141073e0 0x7f8a1410f5b0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f8a0400c890 tx=0x7f8a0400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:27:31.880 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:31.876+0000 7f8a0affd640 1 -- 172.21.15.142:0/703998198 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a04004220 con 0x7f8a141073e0 2024-09-06T14:27:31.880 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:31.876+0000 7f8a0affd640 1 -- 172.21.15.142:0/703998198 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7f8a04007e30 con 0x7f8a141073e0 2024-09-06T14:27:31.880 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:31.876+0000 7f8a1ba96640 1 -- 172.21.15.142:0/703998198 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a141a6d30 con 0x7f8a141073e0 2024-09-06T14:27:31.880 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:31.876+0000 7f8a0affd640 1 -- 172.21.15.142:0/703998198 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a040047a0 con 0x7f8a141073e0 2024-09-06T14:27:31.880 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:31.876+0000 7f8a1ba96640 1 -- 172.21.15.142:0/703998198 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a141a7210 con 0x7f8a141073e0 2024-09-06T14:27:31.880 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:31.877+0000 7f8a1ba96640 1 -- 172.21.15.142:0/703998198 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a14101db0 con 0x7f8a141073e0 2024-09-06T14:27:31.880 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:31.877+0000 7f8a0affd640 1 -- 172.21.15.142:0/703998198 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 22) ==== 99844+0+0 (secure 0 0 0) 0x7f8a040079b0 con 0x7f8a141073e0 2024-09-06T14:27:31.880 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:31.878+0000 7f8a0affd640 1 --2- 172.21.15.142:0/703998198 >> v2:172.21.15.142:6800/2944545612 conn(0x7f8a00077620 0x7f8a00079ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:27:31.880 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:31.878+0000 7f8a19832640 1 --2- 172.21.15.142:0/703998198 >> v2:172.21.15.142:6800/2944545612 conn(0x7f8a00077620 0x7f8a00079ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:27:31.880 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:31.878+0000 7f8a0affd640 1 -- 172.21.15.142:0/703998198 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 6130+0+0 (secure 0 0 0) 0x7f8a040995f0 con 0x7f8a141073e0 2024-09-06T14:27:31.880 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:31.878+0000 7f8a19832640 1 --2- 172.21.15.142:0/703998198 >> v2:172.21.15.142:6800/2944545612 conn(0x7f8a00077620 0x7f8a00079ae0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f8a100095f0 tx=0x7f8a10005f90 comp rx=0 tx=0).ready entity=mgr.14601 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:27:31.881 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:31.882+0000 7f8a0affd640 1 -- 172.21.15.142:0/703998198 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7f8a04014030 con 0x7f8a141073e0 2024-09-06T14:27:32.073 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:32.076+0000 7f8a1ba96640 1 -- 172.21.15.142:0/703998198 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f8a141a7530 con 0x7f8a141073e0 2024-09-06T14:27:32.073 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:32.076+0000 7f8a0affd640 1 -- 172.21.15.142:0/703998198 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+212 (secure 0 0 0) 0x7f8a04066820 con 0x7f8a141073e0 2024-09-06T14:27:32.074 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:27:32.074 INFO:teuthology.orchestra.run.smithi142.stdout:{"status":"HEALTH_WARN","checks":{"MGR_MODULE_DEPENDENCY":{"severity":"HEALTH_WARN","summary":{"message":"Module 'volumes' has failed dependency: No module named 'ceph.fs'","count":1},"muted":false}},"mutes":[]} 2024-09-06T14:27:32.076 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:32.078+0000 7f8a1ba96640 1 -- 172.21.15.142:0/703998198 >> v2:172.21.15.142:6800/2944545612 conn(0x7f8a00077620 msgr2=0x7f8a00079ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:27:32.076 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:32.078+0000 7f8a1ba96640 1 --2- 172.21.15.142:0/703998198 >> v2:172.21.15.142:6800/2944545612 conn(0x7f8a00077620 0x7f8a00079ae0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f8a100095f0 tx=0x7f8a10005f90 comp rx=0 tx=0).stop 2024-09-06T14:27:32.076 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:32.078+0000 7f8a1ba96640 1 -- 172.21.15.142:0/703998198 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8a141073e0 msgr2=0x7f8a1410f5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:27:32.076 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:32.078+0000 7f8a1ba96640 1 --2- 172.21.15.142:0/703998198 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8a141073e0 0x7f8a1410f5b0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f8a0400c890 tx=0x7f8a0400cd50 comp rx=0 tx=0).stop 2024-09-06T14:27:32.076 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:32.079+0000 7f8a1ba96640 1 -- 172.21.15.142:0/703998198 shutdown_connections 2024-09-06T14:27:32.076 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:32.079+0000 7f8a1ba96640 1 --2- 172.21.15.142:0/703998198 >> v2:172.21.15.142:6800/2944545612 conn(0x7f8a00077620 0x7f8a00079ae0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:32.076 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:32.079+0000 7f8a1ba96640 1 --2- 172.21.15.142:0/703998198 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8a14107d60 0x7f8a1410faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:32.076 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:32.079+0000 7f8a1ba96640 1 --2- 172.21.15.142:0/703998198 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8a141073e0 0x7f8a1410f5b0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:32.076 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:32.079+0000 7f8a1ba96640 1 --2- 172.21.15.142:0/703998198 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8a141019d0 0x7f8a1410f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:32.077 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:32.079+0000 7f8a1ba96640 1 -- 172.21.15.142:0/703998198 >> 172.21.15.142:0/703998198 conn(0x7f8a140fd460 msgr2=0x7f8a140fec70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:27:32.077 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:32.079+0000 7f8a1ba96640 1 -- 172.21.15.142:0/703998198 shutdown_connections 2024-09-06T14:27:32.077 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:32.079+0000 7f8a1ba96640 1 -- 172.21.15.142:0/703998198 wait complete. 2024-09-06T14:27:33.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:27:32 smithi142 ceph-mon[29958]: pgmap v83: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:27:33.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:27:32 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/703998198' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:27:33.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:27:32 smithi142 ceph-mon[37372]: pgmap v83: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:27:33.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:27:32 smithi142 ceph-mon[37372]: from='client.? 172.21.15.142:0/703998198' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:27:33.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:27:32 smithi162 ceph-mon[33787]: pgmap v83: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:27:33.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:27:32 smithi162 ceph-mon[33787]: from='client.? 172.21.15.142:0/703998198' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:27:34.709 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:27:34 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: ::ffff:172.21.15.162 - - [06/Sep/2024:14:27:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-06T14:27:35.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:27:34 smithi142 ceph-mon[29958]: pgmap v84: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:27:35.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:27:34 smithi142 ceph-mon[37372]: pgmap v84: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:27:35.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:27:34 smithi162 ceph-mon[33787]: pgmap v84: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:27:35.816 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph health --format=json 2024-09-06T14:27:36.079 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:27:36.559 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:36.525+0000 7f210dc75640 1 -- 172.21.15.142:0/1152106987 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f21081019c0 msgr2=0x7f2108101da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:27:36.559 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:36.525+0000 7f210dc75640 1 --2- 172.21.15.142:0/1152106987 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f21081019c0 0x7f2108101da0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f20fc009860 tx=0x7f20fc02a340 comp rx=0 tx=0).stop 2024-09-06T14:27:36.560 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:36.525+0000 7f210dc75640 1 -- 172.21.15.142:0/1152106987 shutdown_connections 2024-09-06T14:27:36.560 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:36.525+0000 7f210dc75640 1 --2- 172.21.15.142:0/1152106987 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2108107d50 0x7f210810e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:36.560 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:36.525+0000 7f210dc75640 1 --2- 172.21.15.142:0/1152106987 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f21081073d0 0x7f2108107810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:36.560 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:36.525+0000 7f210dc75640 1 --2- 172.21.15.142:0/1152106987 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f21081019c0 0x7f2108101da0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:36.560 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:36.525+0000 7f210dc75640 1 -- 172.21.15.142:0/1152106987 >> 172.21.15.142:0/1152106987 conn(0x7f21080fd430 msgr2=0x7f21080ff850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:27:36.560 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:36.525+0000 7f210dc75640 1 -- 172.21.15.142:0/1152106987 shutdown_connections 2024-09-06T14:27:36.560 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:36.525+0000 7f210dc75640 1 -- 172.21.15.142:0/1152106987 wait complete. 2024-09-06T14:27:36.560 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:36.526+0000 7f210dc75640 1 Processor -- start 2024-09-06T14:27:36.560 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:36.526+0000 7f210dc75640 1 -- start start 2024-09-06T14:27:36.560 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:36.526+0000 7f210dc75640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f21081019c0 0x7f2108198660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:27:36.560 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:36.526+0000 7f210dc75640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f21081073d0 0x7f2108198ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:27:36.560 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:36.526+0000 7f210dc75640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2108107d50 0x7f210819df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:27:36.560 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:36.526+0000 7f210dc75640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f21081110e0 con 0x7f21081073d0 2024-09-06T14:27:36.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:36.526+0000 7f210dc75640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f2108110f60 con 0x7f21081019c0 2024-09-06T14:27:36.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:36.526+0000 7f210dc75640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7f2108111260 con 0x7f2108107d50 2024-09-06T14:27:36.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:36.526+0000 7f2107fff640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2108107d50 0x7f210819df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:27:36.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:36.526+0000 7f2106ffd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f21081073d0 0x7f2108198ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:27:36.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:36.526+0000 7f2106ffd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f21081073d0 0x7f2108198ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:50094/0 (socket says 172.21.15.142:50094) 2024-09-06T14:27:36.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:36.526+0000 7f2106ffd640 1 -- 172.21.15.142:0/3300273168 learned_addr learned my addr 172.21.15.142:0/3300273168 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:27:36.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:36.526+0000 7f21077fe640 1 --2- 172.21.15.142:0/3300273168 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f21081019c0 0x7f2108198660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:27:36.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:36.526+0000 7f2106ffd640 1 -- 172.21.15.142:0/3300273168 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2108107d50 msgr2=0x7f210819df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:27:36.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:36.526+0000 7f2106ffd640 1 --2- 172.21.15.142:0/3300273168 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2108107d50 0x7f210819df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:36.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:36.526+0000 7f2106ffd640 1 -- 172.21.15.142:0/3300273168 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f21081019c0 msgr2=0x7f2108198660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:27:36.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:36.526+0000 7f2106ffd640 1 --2- 172.21.15.142:0/3300273168 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f21081019c0 0x7f2108198660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:36.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:36.526+0000 7f2106ffd640 1 -- 172.21.15.142:0/3300273168 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f210819e710 con 0x7f21081073d0 2024-09-06T14:27:36.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:36.527+0000 7f2107fff640 1 --2- 172.21.15.142:0/3300273168 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2108107d50 0x7f210819df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-06T14:27:36.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:36.527+0000 7f2106ffd640 1 --2- 172.21.15.142:0/3300273168 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f21081073d0 0x7f2108198ba0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f20f800e9c0 tx=0x7f20f800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:27:36.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:36.527+0000 7f2104ff9640 1 -- 172.21.15.142:0/3300273168 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20f800cde0 con 0x7f21081073d0 2024-09-06T14:27:36.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:36.527+0000 7f2104ff9640 1 -- 172.21.15.142:0/3300273168 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7f20f80035f0 con 0x7f21081073d0 2024-09-06T14:27:36.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:36.527+0000 7f210dc75640 1 -- 172.21.15.142:0/3300273168 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f210819ea00 con 0x7f21081073d0 2024-09-06T14:27:36.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:36.527+0000 7f2104ff9640 1 -- 172.21.15.142:0/3300273168 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20f80044a0 con 0x7f21081073d0 2024-09-06T14:27:36.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:36.527+0000 7f210dc75640 1 -- 172.21.15.142:0/3300273168 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21081a5290 con 0x7f21081073d0 2024-09-06T14:27:36.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:36.528+0000 7f210dc75640 1 -- 172.21.15.142:0/3300273168 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2108101da0 con 0x7f21081073d0 2024-09-06T14:27:36.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:36.529+0000 7f2104ff9640 1 -- 172.21.15.142:0/3300273168 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 22) ==== 99844+0+0 (secure 0 0 0) 0x7f20f8004000 con 0x7f21081073d0 2024-09-06T14:27:36.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:36.529+0000 7f2104ff9640 1 --2- 172.21.15.142:0/3300273168 >> v2:172.21.15.142:6800/2944545612 conn(0x7f20f0077620 0x7f20f0079ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:27:36.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:36.529+0000 7f2104ff9640 1 -- 172.21.15.142:0/3300273168 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 6130+0+0 (secure 0 0 0) 0x7f20f8099be0 con 0x7f21081073d0 2024-09-06T14:27:36.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:36.529+0000 7f21077fe640 1 --2- 172.21.15.142:0/3300273168 >> v2:172.21.15.142:6800/2944545612 conn(0x7f20f0077620 0x7f20f0079ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:27:36.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:36.530+0000 7f21077fe640 1 --2- 172.21.15.142:0/3300273168 >> v2:172.21.15.142:6800/2944545612 conn(0x7f20f0077620 0x7f20f0079ae0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f20fc02a930 tx=0x7f20fc0058d0 comp rx=0 tx=0).ready entity=mgr.14601 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:27:36.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:36.533+0000 7f2104ff9640 1 -- 172.21.15.142:0/3300273168 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7f20f8066e90 con 0x7f21081073d0 2024-09-06T14:27:36.718 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:36.721+0000 7f210dc75640 1 -- 172.21.15.142:0/3300273168 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f21081a5460 con 0x7f21081073d0 2024-09-06T14:27:36.718 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:36.721+0000 7f2104ff9640 1 -- 172.21.15.142:0/3300273168 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+212 (secure 0 0 0) 0x7f20f806b1d0 con 0x7f21081073d0 2024-09-06T14:27:36.718 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:27:36.718 INFO:teuthology.orchestra.run.smithi142.stdout:{"status":"HEALTH_WARN","checks":{"MGR_MODULE_DEPENDENCY":{"severity":"HEALTH_WARN","summary":{"message":"Module 'volumes' has failed dependency: No module named 'ceph.fs'","count":1},"muted":false}},"mutes":[]} 2024-09-06T14:27:36.720 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:36.723+0000 7f210dc75640 1 -- 172.21.15.142:0/3300273168 >> v2:172.21.15.142:6800/2944545612 conn(0x7f20f0077620 msgr2=0x7f20f0079ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:27:36.720 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:36.723+0000 7f210dc75640 1 --2- 172.21.15.142:0/3300273168 >> v2:172.21.15.142:6800/2944545612 conn(0x7f20f0077620 0x7f20f0079ae0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f20fc02a930 tx=0x7f20fc0058d0 comp rx=0 tx=0).stop 2024-09-06T14:27:36.720 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:36.723+0000 7f210dc75640 1 -- 172.21.15.142:0/3300273168 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f21081073d0 msgr2=0x7f2108198ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:27:36.720 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:36.723+0000 7f210dc75640 1 --2- 172.21.15.142:0/3300273168 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f21081073d0 0x7f2108198ba0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f20f800e9c0 tx=0x7f20f800ee80 comp rx=0 tx=0).stop 2024-09-06T14:27:36.720 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:36.723+0000 7f210dc75640 1 -- 172.21.15.142:0/3300273168 shutdown_connections 2024-09-06T14:27:36.720 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:36.723+0000 7f210dc75640 1 --2- 172.21.15.142:0/3300273168 >> v2:172.21.15.142:6800/2944545612 conn(0x7f20f0077620 0x7f20f0079ae0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:36.721 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:36.723+0000 7f210dc75640 1 --2- 172.21.15.142:0/3300273168 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2108107d50 0x7f210819df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:36.721 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:36.723+0000 7f210dc75640 1 --2- 172.21.15.142:0/3300273168 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f21081073d0 0x7f2108198ba0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:36.721 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:36.723+0000 7f210dc75640 1 --2- 172.21.15.142:0/3300273168 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f21081019c0 0x7f2108198660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:36.721 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:36.723+0000 7f210dc75640 1 -- 172.21.15.142:0/3300273168 >> 172.21.15.142:0/3300273168 conn(0x7f21080fd430 msgr2=0x7f210806a7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:27:36.721 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:36.723+0000 7f210dc75640 1 -- 172.21.15.142:0/3300273168 shutdown_connections 2024-09-06T14:27:36.721 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:36.723+0000 7f210dc75640 1 -- 172.21.15.142:0/3300273168 wait complete. 2024-09-06T14:27:37.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:27:36 smithi142 ceph-mon[29958]: pgmap v85: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:27:37.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:27:36 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/3300273168' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:27:37.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:27:36 smithi142 ceph-mon[37372]: pgmap v85: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:27:37.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:27:36 smithi142 ceph-mon[37372]: from='client.? 172.21.15.142:0/3300273168' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:27:37.297 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 06 14:27:37 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-iscsi-iscsi-a[59853]: debug there is no tcmu-runner data available 2024-09-06T14:27:37.298 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:27:36 smithi162 ceph-mon[33787]: pgmap v85: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:27:37.298 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:27:36 smithi162 ceph-mon[33787]: from='client.? 172.21.15.142:0/3300273168' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:27:39.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:27:38 smithi142 ceph-mon[29958]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-06T14:27:39.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:27:38 smithi142 ceph-mon[29958]: pgmap v86: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:27:39.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:27:38 smithi142 ceph-mon[37372]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-06T14:27:39.210 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:27:38 smithi142 ceph-mon[37372]: pgmap v86: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:27:39.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:27:38 smithi162 ceph-mon[33787]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-06T14:27:39.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:27:38 smithi162 ceph-mon[33787]: pgmap v86: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:27:40.462 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph health --format=json 2024-09-06T14:27:40.721 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:27:41.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:27:40 smithi142 ceph-mon[37372]: pgmap v87: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:27:41.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:27:40 smithi142 ceph-mon[37372]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-06T14:27:41.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:27:40 smithi142 ceph-mon[29958]: pgmap v87: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:27:41.210 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:27:40 smithi142 ceph-mon[29958]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-06T14:27:41.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:27:40 smithi162 ceph-mon[33787]: pgmap v87: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:27:41.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:27:40 smithi162 ceph-mon[33787]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-06T14:27:41.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:41.357+0000 7faed6bbb640 1 -- 172.21.15.142:0/140253998 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faed0108520 msgr2=0x7faed0108960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:27:41.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:41.357+0000 7faed6bbb640 1 --2- 172.21.15.142:0/140253998 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faed0108520 0x7faed0108960 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7faebc009860 tx=0x7faebc02a340 comp rx=0 tx=0).stop 2024-09-06T14:27:41.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:41.357+0000 7faed6bbb640 1 -- 172.21.15.142:0/140253998 shutdown_connections 2024-09-06T14:27:41.360 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:41.357+0000 7faed6bbb640 1 --2- 172.21.15.142:0/140253998 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faed0108ea0 0x7faed010f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:41.360 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:41.357+0000 7faed6bbb640 1 --2- 172.21.15.142:0/140253998 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faed0108520 0x7faed0108960 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:41.360 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:41.357+0000 7faed6bbb640 1 --2- 172.21.15.142:0/140253998 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7faed0102b10 0x7faed0102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:41.360 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:41.357+0000 7faed6bbb640 1 -- 172.21.15.142:0/140253998 >> 172.21.15.142:0/140253998 conn(0x7faed00fe5a0 msgr2=0x7faed01009c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:27:41.360 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:41.357+0000 7faed6bbb640 1 -- 172.21.15.142:0/140253998 shutdown_connections 2024-09-06T14:27:41.360 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:41.357+0000 7faed6bbb640 1 -- 172.21.15.142:0/140253998 wait complete. 2024-09-06T14:27:41.360 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:41.358+0000 7faed6bbb640 1 Processor -- start 2024-09-06T14:27:41.360 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:41.358+0000 7faed6bbb640 1 -- start start 2024-09-06T14:27:41.360 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:41.358+0000 7faed6bbb640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faed0102b10 0x7faed01986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:27:41.360 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:41.358+0000 7faed6bbb640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faed0108520 0x7faed0198be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:27:41.360 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:41.358+0000 7faed6bbb640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7faed0108ea0 0x7faed019df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:27:41.360 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:41.358+0000 7faed6bbb640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7faed01121f0 con 0x7faed0102b10 2024-09-06T14:27:41.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:41.358+0000 7faed6bbb640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7faed0112070 con 0x7faed0108520 2024-09-06T14:27:41.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:41.358+0000 7faed6bbb640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7faed0112370 con 0x7faed0108ea0 2024-09-06T14:27:41.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:41.358+0000 7faed4957640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faed0102b10 0x7faed01986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:27:41.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:41.358+0000 7faed5158640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7faed0108ea0 0x7faed019df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:27:41.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:41.358+0000 7faed4957640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faed0102b10 0x7faed01986a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:47888/0 (socket says 172.21.15.142:47888) 2024-09-06T14:27:41.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:41.358+0000 7faed5158640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7faed0108ea0 0x7faed019df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:60210/0 (socket says 172.21.15.142:60210) 2024-09-06T14:27:41.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:41.358+0000 7faed4957640 1 -- 172.21.15.142:0/1111762314 learned_addr learned my addr 172.21.15.142:0/1111762314 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:27:41.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:41.358+0000 7faecffff640 1 --2- 172.21.15.142:0/1111762314 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faed0108520 0x7faed0198be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:27:41.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:41.359+0000 7faed4957640 1 -- 172.21.15.142:0/1111762314 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7faed0108ea0 msgr2=0x7faed019df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:27:41.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:41.359+0000 7faed4957640 1 --2- 172.21.15.142:0/1111762314 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7faed0108ea0 0x7faed019df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:41.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:41.359+0000 7faed4957640 1 -- 172.21.15.142:0/1111762314 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faed0108520 msgr2=0x7faed0198be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:27:41.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:41.359+0000 7faed4957640 1 --2- 172.21.15.142:0/1111762314 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faed0108520 0x7faed0198be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:41.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:41.359+0000 7faed4957640 1 -- 172.21.15.142:0/1111762314 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faed019e750 con 0x7faed0102b10 2024-09-06T14:27:41.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:41.359+0000 7faed5158640 1 --2- 172.21.15.142:0/1111762314 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7faed0108ea0 0x7faed019df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-06T14:27:41.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:41.359+0000 7faed4957640 1 --2- 172.21.15.142:0/1111762314 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faed0102b10 0x7faed01986a0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7faec400e9c0 tx=0x7faec400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:27:41.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:41.359+0000 7faecdffb640 1 -- 172.21.15.142:0/1111762314 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faec400cde0 con 0x7faed0102b10 2024-09-06T14:27:41.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:41.359+0000 7faecdffb640 1 -- 172.21.15.142:0/1111762314 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7faec40035f0 con 0x7faed0102b10 2024-09-06T14:27:41.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:41.359+0000 7faed6bbb640 1 -- 172.21.15.142:0/1111762314 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faed019ea40 con 0x7faed0102b10 2024-09-06T14:27:41.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:41.359+0000 7faecdffb640 1 -- 172.21.15.142:0/1111762314 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faec40044a0 con 0x7faed0102b10 2024-09-06T14:27:41.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:41.359+0000 7faed6bbb640 1 -- 172.21.15.142:0/1111762314 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faed01a52d0 con 0x7faed0102b10 2024-09-06T14:27:41.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:41.360+0000 7faed6bbb640 1 -- 172.21.15.142:0/1111762314 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faed0102ef0 con 0x7faed0102b10 2024-09-06T14:27:41.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:41.361+0000 7faecdffb640 1 -- 172.21.15.142:0/1111762314 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 22) ==== 99844+0+0 (secure 0 0 0) 0x7faec4004000 con 0x7faed0102b10 2024-09-06T14:27:41.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:41.361+0000 7faecdffb640 1 --2- 172.21.15.142:0/1111762314 >> v2:172.21.15.142:6800/2944545612 conn(0x7faec0077620 0x7faec0079ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:27:41.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:41.361+0000 7faecdffb640 1 -- 172.21.15.142:0/1111762314 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 6130+0+0 (secure 0 0 0) 0x7faec4099be0 con 0x7faed0102b10 2024-09-06T14:27:41.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:41.361+0000 7faecffff640 1 --2- 172.21.15.142:0/1111762314 >> v2:172.21.15.142:6800/2944545612 conn(0x7faec0077620 0x7faec0079ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:27:41.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:41.362+0000 7faecffff640 1 --2- 172.21.15.142:0/1111762314 >> v2:172.21.15.142:6800/2944545612 conn(0x7faec0077620 0x7faec0079ae0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7faebc02a930 tx=0x7faebc0058d0 comp rx=0 tx=0).ready entity=mgr.14601 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:27:41.363 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:41.365+0000 7faecdffb640 1 -- 172.21.15.142:0/1111762314 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7faec4066e90 con 0x7faed0102b10 2024-09-06T14:27:41.553 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:41.555+0000 7faed6bbb640 1 -- 172.21.15.142:0/1111762314 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7faed010ac80 con 0x7faed0102b10 2024-09-06T14:27:41.553 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:41.556+0000 7faecdffb640 1 -- 172.21.15.142:0/1111762314 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+212 (secure 0 0 0) 0x7faec406b1d0 con 0x7faed0102b10 2024-09-06T14:27:41.553 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:27:41.553 INFO:teuthology.orchestra.run.smithi142.stdout:{"status":"HEALTH_WARN","checks":{"MGR_MODULE_DEPENDENCY":{"severity":"HEALTH_WARN","summary":{"message":"Module 'volumes' has failed dependency: No module named 'ceph.fs'","count":1},"muted":false}},"mutes":[]} 2024-09-06T14:27:41.555 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:41.558+0000 7faed6bbb640 1 -- 172.21.15.142:0/1111762314 >> v2:172.21.15.142:6800/2944545612 conn(0x7faec0077620 msgr2=0x7faec0079ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:27:41.555 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:41.558+0000 7faed6bbb640 1 --2- 172.21.15.142:0/1111762314 >> v2:172.21.15.142:6800/2944545612 conn(0x7faec0077620 0x7faec0079ae0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7faebc02a930 tx=0x7faebc0058d0 comp rx=0 tx=0).stop 2024-09-06T14:27:41.555 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:41.558+0000 7faed6bbb640 1 -- 172.21.15.142:0/1111762314 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faed0102b10 msgr2=0x7faed01986a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:27:41.556 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:41.558+0000 7faed6bbb640 1 --2- 172.21.15.142:0/1111762314 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faed0102b10 0x7faed01986a0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7faec400e9c0 tx=0x7faec400ee80 comp rx=0 tx=0).stop 2024-09-06T14:27:41.556 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:41.558+0000 7faed6bbb640 1 -- 172.21.15.142:0/1111762314 shutdown_connections 2024-09-06T14:27:41.556 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:41.558+0000 7faed6bbb640 1 --2- 172.21.15.142:0/1111762314 >> v2:172.21.15.142:6800/2944545612 conn(0x7faec0077620 0x7faec0079ae0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:41.556 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:41.558+0000 7faed6bbb640 1 --2- 172.21.15.142:0/1111762314 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7faed0108ea0 0x7faed019df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:41.556 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:41.558+0000 7faed6bbb640 1 --2- 172.21.15.142:0/1111762314 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faed0108520 0x7faed0198be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:41.556 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:41.558+0000 7faed6bbb640 1 --2- 172.21.15.142:0/1111762314 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faed0102b10 0x7faed01986a0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:41.556 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:41.558+0000 7faed6bbb640 1 -- 172.21.15.142:0/1111762314 >> 172.21.15.142:0/1111762314 conn(0x7faed00fe5a0 msgr2=0x7faed0100990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:27:41.556 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:41.558+0000 7faed6bbb640 1 -- 172.21.15.142:0/1111762314 shutdown_connections 2024-09-06T14:27:41.556 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:41.558+0000 7faed6bbb640 1 -- 172.21.15.142:0/1111762314 wait complete. 2024-09-06T14:27:42.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:27:41 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/1111762314' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:27:42.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:27:41 smithi142 ceph-mon[37372]: from='client.? 172.21.15.142:0/1111762314' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:27:42.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:27:41 smithi162 ceph-mon[33787]: from='client.? 172.21.15.142:0/1111762314' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:27:43.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:27:42 smithi162 ceph-mon[33787]: pgmap v88: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:27:43.459 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:27:42 smithi142 ceph-mon[29958]: pgmap v88: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:27:43.459 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:27:42 smithi142 ceph-mon[37372]: pgmap v88: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:27:44.709 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:27:44 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: ::ffff:172.21.15.162 - - [06/Sep/2024:14:27:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-06T14:27:45.246 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph health --format=json 2024-09-06T14:27:45.273 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:27:44 smithi142 ceph-mon[29958]: pgmap v89: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:27:45.273 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:27:44 smithi142 ceph-mon[37372]: pgmap v89: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:27:45.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:27:44 smithi162 ceph-mon[33787]: pgmap v89: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:27:45.505 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:27:45.984 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:45.982+0000 7fac0a569640 1 -- 172.21.15.142:0/1515363348 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fac0410ce30 msgr2=0x7fac0410f2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:27:45.984 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:45.982+0000 7fac0a569640 1 --2- 172.21.15.142:0/1515363348 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fac0410ce30 0x7fac0410f2a0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fabfc009ca0 tx=0x7fabfc02a8c0 comp rx=0 tx=0).stop 2024-09-06T14:27:45.985 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:45.982+0000 7fac0a569640 1 -- 172.21.15.142:0/1515363348 shutdown_connections 2024-09-06T14:27:45.985 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:45.982+0000 7fac0a569640 1 --2- 172.21.15.142:0/1515363348 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fac0410ce30 0x7fac0410f2a0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:45.985 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:45.982+0000 7fac0a569640 1 --2- 172.21.15.142:0/1515363348 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fac04104280 0x7fac0410c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:45.985 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:45.982+0000 7fac0a569640 1 --2- 172.21.15.142:0/1515363348 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fac04103880 0x7fac04103c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:45.985 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:45.982+0000 7fac0a569640 1 -- 172.21.15.142:0/1515363348 >> 172.21.15.142:0/1515363348 conn(0x7fac040fd450 msgr2=0x7fac040ff870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:27:45.985 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:45.982+0000 7fac0a569640 1 -- 172.21.15.142:0/1515363348 shutdown_connections 2024-09-06T14:27:45.985 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:45.982+0000 7fac0a569640 1 -- 172.21.15.142:0/1515363348 wait complete. 2024-09-06T14:27:45.985 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:45.983+0000 7fac0a569640 1 Processor -- start 2024-09-06T14:27:45.985 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:45.983+0000 7fac0a569640 1 -- start start 2024-09-06T14:27:45.985 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:45.983+0000 7fac0a569640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fac04103880 0x7fac041985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:27:45.985 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:45.983+0000 7fac0a569640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fac04104280 0x7fac04198b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:27:45.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:45.983+0000 7fac0a569640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fac0410ce30 0x7fac0419deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:27:45.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:45.983+0000 7fac0a569640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fac041121c0 con 0x7fac04103880 2024-09-06T14:27:45.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:45.983+0000 7fac0a569640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fac04112040 con 0x7fac04104280 2024-09-06T14:27:45.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:45.983+0000 7fac0a569640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7fac04112340 con 0x7fac0410ce30 2024-09-06T14:27:45.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:45.983+0000 7fac03fff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fac04103880 0x7fac041985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:27:45.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:45.983+0000 7fac03fff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fac04103880 0x7fac041985d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:47904/0 (socket says 172.21.15.142:47904) 2024-09-06T14:27:45.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:45.983+0000 7fac03fff640 1 -- 172.21.15.142:0/4276314031 learned_addr learned my addr 172.21.15.142:0/4276314031 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:27:45.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:45.983+0000 7fac08b06640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fac0410ce30 0x7fac0419deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:27:45.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:45.984+0000 7fac037fe640 1 --2- 172.21.15.142:0/4276314031 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fac04104280 0x7fac04198b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:27:45.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:45.984+0000 7fac03fff640 1 -- 172.21.15.142:0/4276314031 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fac0410ce30 msgr2=0x7fac0419deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:27:45.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:45.984+0000 7fac03fff640 1 --2- 172.21.15.142:0/4276314031 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fac0410ce30 0x7fac0419deb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:45.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:45.984+0000 7fac03fff640 1 -- 172.21.15.142:0/4276314031 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fac04104280 msgr2=0x7fac04198b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:27:45.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:45.984+0000 7fac03fff640 1 --2- 172.21.15.142:0/4276314031 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fac04104280 0x7fac04198b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:45.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:45.984+0000 7fac03fff640 1 -- 172.21.15.142:0/4276314031 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac0419e680 con 0x7fac04103880 2024-09-06T14:27:45.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:45.984+0000 7fac08b06640 1 --2- 172.21.15.142:0/4276314031 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fac0410ce30 0x7fac0419deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-06T14:27:45.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:45.984+0000 7fac03fff640 1 --2- 172.21.15.142:0/4276314031 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fac04103880 0x7fac041985d0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fabf400ea60 tx=0x7fabf400ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:27:45.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:45.984+0000 7fac017fa640 1 -- 172.21.15.142:0/4276314031 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabf400cde0 con 0x7fac04103880 2024-09-06T14:27:45.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:45.984+0000 7fac0a569640 1 -- 172.21.15.142:0/4276314031 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac0419e970 con 0x7fac04103880 2024-09-06T14:27:45.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:45.984+0000 7fac017fa640 1 -- 172.21.15.142:0/4276314031 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7fabf40035f0 con 0x7fac04103880 2024-09-06T14:27:45.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:45.984+0000 7fac017fa640 1 -- 172.21.15.142:0/4276314031 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabf40044a0 con 0x7fac04103880 2024-09-06T14:27:45.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:45.984+0000 7fac0a569640 1 -- 172.21.15.142:0/4276314031 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac041a5200 con 0x7fac04103880 2024-09-06T14:27:45.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:45.985+0000 7fac0a569640 1 -- 172.21.15.142:0/4276314031 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac04103c60 con 0x7fac04103880 2024-09-06T14:27:45.988 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:45.986+0000 7fac017fa640 1 -- 172.21.15.142:0/4276314031 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 22) ==== 99844+0+0 (secure 0 0 0) 0x7fabf4004000 con 0x7fac04103880 2024-09-06T14:27:45.988 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:45.986+0000 7fac017fa640 1 --2- 172.21.15.142:0/4276314031 >> v2:172.21.15.142:6800/2944545612 conn(0x7fabec077620 0x7fabec079ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:27:45.988 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:45.986+0000 7fac037fe640 1 --2- 172.21.15.142:0/4276314031 >> v2:172.21.15.142:6800/2944545612 conn(0x7fabec077620 0x7fabec079ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:27:45.988 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:45.986+0000 7fac017fa640 1 -- 172.21.15.142:0/4276314031 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 6130+0+0 (secure 0 0 0) 0x7fabf4099d80 con 0x7fac04103880 2024-09-06T14:27:45.988 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:45.987+0000 7fac037fe640 1 --2- 172.21.15.142:0/4276314031 >> v2:172.21.15.142:6800/2944545612 conn(0x7fabec077620 0x7fabec079ae0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fabf8005e10 tx=0x7fabf8005da0 comp rx=0 tx=0).ready entity=mgr.14601 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:27:45.988 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:45.990+0000 7fac017fa640 1 -- 172.21.15.142:0/4276314031 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7fabf4062e40 con 0x7fac04103880 2024-09-06T14:27:46.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:46.183+0000 7fac0a569640 1 -- 172.21.15.142:0/4276314031 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fac041a53d0 con 0x7fac04103880 2024-09-06T14:27:46.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:46.184+0000 7fac017fa640 1 -- 172.21.15.142:0/4276314031 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+212 (secure 0 0 0) 0x7fabf4083e20 con 0x7fac04103880 2024-09-06T14:27:46.181 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:27:46.181 INFO:teuthology.orchestra.run.smithi142.stdout:{"status":"HEALTH_WARN","checks":{"MGR_MODULE_DEPENDENCY":{"severity":"HEALTH_WARN","summary":{"message":"Module 'volumes' has failed dependency: No module named 'ceph.fs'","count":1},"muted":false}},"mutes":[]} 2024-09-06T14:27:46.183 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:46.186+0000 7fac0a569640 1 -- 172.21.15.142:0/4276314031 >> v2:172.21.15.142:6800/2944545612 conn(0x7fabec077620 msgr2=0x7fabec079ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:27:46.183 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:46.186+0000 7fac0a569640 1 --2- 172.21.15.142:0/4276314031 >> v2:172.21.15.142:6800/2944545612 conn(0x7fabec077620 0x7fabec079ae0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fabf8005e10 tx=0x7fabf8005da0 comp rx=0 tx=0).stop 2024-09-06T14:27:46.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:46.186+0000 7fac0a569640 1 -- 172.21.15.142:0/4276314031 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fac04103880 msgr2=0x7fac041985d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:27:46.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:46.186+0000 7fac0a569640 1 --2- 172.21.15.142:0/4276314031 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fac04103880 0x7fac041985d0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fabf400ea60 tx=0x7fabf400ef20 comp rx=0 tx=0).stop 2024-09-06T14:27:46.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:46.186+0000 7fac0a569640 1 -- 172.21.15.142:0/4276314031 shutdown_connections 2024-09-06T14:27:46.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:46.186+0000 7fac0a569640 1 --2- 172.21.15.142:0/4276314031 >> v2:172.21.15.142:6800/2944545612 conn(0x7fabec077620 0x7fabec079ae0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:46.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:46.186+0000 7fac0a569640 1 --2- 172.21.15.142:0/4276314031 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fac0410ce30 0x7fac0419deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:46.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:46.186+0000 7fac0a569640 1 --2- 172.21.15.142:0/4276314031 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fac04104280 0x7fac04198b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:46.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:46.186+0000 7fac0a569640 1 --2- 172.21.15.142:0/4276314031 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fac04103880 0x7fac041985d0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:46.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:46.186+0000 7fac0a569640 1 -- 172.21.15.142:0/4276314031 >> 172.21.15.142:0/4276314031 conn(0x7fac040fd450 msgr2=0x7fac0410f070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:27:46.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:46.186+0000 7fac0a569640 1 -- 172.21.15.142:0/4276314031 shutdown_connections 2024-09-06T14:27:46.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:46.186+0000 7fac0a569640 1 -- 172.21.15.142:0/4276314031 wait complete. 2024-09-06T14:27:47.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:27:46 smithi142 ceph-mon[29958]: pgmap v90: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:27:47.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:27:46 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/4276314031' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:27:47.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:27:46 smithi142 ceph-mon[37372]: pgmap v90: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:27:47.210 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:27:46 smithi142 ceph-mon[37372]: from='client.? 172.21.15.142:0/4276314031' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:27:47.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:27:46 smithi162 ceph-mon[33787]: pgmap v90: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:27:47.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:27:46 smithi162 ceph-mon[33787]: from='client.? 172.21.15.142:0/4276314031' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:27:47.298 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 06 14:27:47 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-iscsi-iscsi-a[59853]: debug there is no tcmu-runner data available 2024-09-06T14:27:49.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:27:48 smithi162 ceph-mon[33787]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-06T14:27:49.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:27:48 smithi162 ceph-mon[33787]: pgmap v91: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:27:49.459 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:27:48 smithi142 ceph-mon[29958]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-06T14:27:49.459 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:27:48 smithi142 ceph-mon[29958]: pgmap v91: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:27:49.460 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:27:48 smithi142 ceph-mon[37372]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-06T14:27:49.460 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:27:48 smithi142 ceph-mon[37372]: pgmap v91: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:27:49.917 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph health --format=json 2024-09-06T14:27:50.181 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:27:50.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:50.605+0000 7f3f65b64640 1 -- 172.21.15.142:0/6867260 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3f60102ae0 msgr2=0x7f3f60102ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:27:50.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:50.605+0000 7f3f65b64640 1 --2- 172.21.15.142:0/6867260 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3f60102ae0 0x7f3f60102ec0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f3f54009900 tx=0x7f3f5402a3f0 comp rx=0 tx=0).stop 2024-09-06T14:27:50.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:50.605+0000 7f3f65b64640 1 -- 172.21.15.142:0/6867260 shutdown_connections 2024-09-06T14:27:50.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:50.605+0000 7f3f65b64640 1 --2- 172.21.15.142:0/6867260 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3f60108e70 0x7f3f6010f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:50.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:50.605+0000 7f3f65b64640 1 --2- 172.21.15.142:0/6867260 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3f601084f0 0x7f3f60108930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:50.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:50.605+0000 7f3f65b64640 1 --2- 172.21.15.142:0/6867260 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3f60102ae0 0x7f3f60102ec0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:50.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:50.605+0000 7f3f65b64640 1 -- 172.21.15.142:0/6867260 >> 172.21.15.142:0/6867260 conn(0x7f3f600fe530 msgr2=0x7f3f60100950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:27:50.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:50.605+0000 7f3f65b64640 1 -- 172.21.15.142:0/6867260 shutdown_connections 2024-09-06T14:27:50.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:50.605+0000 7f3f65b64640 1 -- 172.21.15.142:0/6867260 wait complete. 2024-09-06T14:27:50.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:50.606+0000 7f3f65b64640 1 Processor -- start 2024-09-06T14:27:50.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:50.606+0000 7f3f65b64640 1 -- start start 2024-09-06T14:27:50.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:50.606+0000 7f3f65b64640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3f60102ae0 0x7f3f60075200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:27:50.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:50.606+0000 7f3f65b64640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3f601084f0 0x7f3f60075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:27:50.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:50.606+0000 7f3f65b64640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3f60108e70 0x7f3f60075c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:27:50.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:50.606+0000 7f3f65b64640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f3f60112240 con 0x7f3f60102ae0 2024-09-06T14:27:50.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:50.606+0000 7f3f65b64640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f3f601120c0 con 0x7f3f60108e70 2024-09-06T14:27:50.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:50.606+0000 7f3f65b64640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7f3f601123c0 con 0x7f3f601084f0 2024-09-06T14:27:50.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:50.606+0000 7f3f5effd640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3f601084f0 0x7f3f60075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:27:50.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:50.606+0000 7f3f5f7fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3f60102ae0 0x7f3f60075200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:27:50.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:50.606+0000 7f3f5effd640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3f601084f0 0x7f3f60075740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:36824/0 (socket says 172.21.15.142:36824) 2024-09-06T14:27:50.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:50.606+0000 7f3f5f7fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3f60102ae0 0x7f3f60075200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:43108/0 (socket says 172.21.15.142:43108) 2024-09-06T14:27:50.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:50.606+0000 7f3f5effd640 1 -- 172.21.15.142:0/964846669 learned_addr learned my addr 172.21.15.142:0/964846669 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:27:50.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:50.606+0000 7f3f5ffff640 1 --2- 172.21.15.142:0/964846669 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3f60108e70 0x7f3f60075c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:27:50.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:50.607+0000 7f3f5f7fe640 1 -- 172.21.15.142:0/964846669 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3f601084f0 msgr2=0x7f3f60075740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:27:50.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:50.607+0000 7f3f5f7fe640 1 --2- 172.21.15.142:0/964846669 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3f601084f0 0x7f3f60075740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:50.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:50.607+0000 7f3f5f7fe640 1 -- 172.21.15.142:0/964846669 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3f60108e70 msgr2=0x7f3f60075c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:27:50.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:50.607+0000 7f3f5f7fe640 1 --2- 172.21.15.142:0/964846669 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3f60108e70 0x7f3f60075c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:50.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:50.607+0000 7f3f5f7fe640 1 -- 172.21.15.142:0/964846669 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f601a6ad0 con 0x7f3f60102ae0 2024-09-06T14:27:50.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:50.607+0000 7f3f5effd640 1 --2- 172.21.15.142:0/964846669 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3f601084f0 0x7f3f60075740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-06T14:27:50.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:50.607+0000 7f3f5f7fe640 1 --2- 172.21.15.142:0/964846669 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3f60102ae0 0x7f3f60075200 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f3f5402a910 tx=0x7f3f540329c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:27:50.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:50.607+0000 7f3f5cff9640 1 -- 172.21.15.142:0/964846669 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f5403f030 con 0x7f3f60102ae0 2024-09-06T14:27:50.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:50.607+0000 7f3f65b64640 1 -- 172.21.15.142:0/964846669 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f601a6d60 con 0x7f3f60102ae0 2024-09-06T14:27:50.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:50.607+0000 7f3f5cff9640 1 -- 172.21.15.142:0/964846669 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7f3f540044c0 con 0x7f3f60102ae0 2024-09-06T14:27:50.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:50.607+0000 7f3f5cff9640 1 -- 172.21.15.142:0/964846669 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f54035070 con 0x7f3f60102ae0 2024-09-06T14:27:50.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:50.607+0000 7f3f65b64640 1 -- 172.21.15.142:0/964846669 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f601a7230 con 0x7f3f60102ae0 2024-09-06T14:27:50.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:50.608+0000 7f3f65b64640 1 -- 172.21.15.142:0/964846669 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f60102ec0 con 0x7f3f60102ae0 2024-09-06T14:27:50.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:50.609+0000 7f3f5cff9640 1 -- 172.21.15.142:0/964846669 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 22) ==== 99844+0+0 (secure 0 0 0) 0x7f3f54042030 con 0x7f3f60102ae0 2024-09-06T14:27:50.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:50.609+0000 7f3f5cff9640 1 --2- 172.21.15.142:0/964846669 >> v2:172.21.15.142:6800/2944545612 conn(0x7f3f48077620 0x7f3f48079ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:27:50.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:50.609+0000 7f3f5effd640 1 --2- 172.21.15.142:0/964846669 >> v2:172.21.15.142:6800/2944545612 conn(0x7f3f48077620 0x7f3f48079ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:27:50.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:50.609+0000 7f3f5cff9640 1 -- 172.21.15.142:0/964846669 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 6130+0+0 (secure 0 0 0) 0x7f3f540b9ce0 con 0x7f3f60102ae0 2024-09-06T14:27:50.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:50.610+0000 7f3f5effd640 1 --2- 172.21.15.142:0/964846669 >> v2:172.21.15.142:6800/2944545612 conn(0x7f3f48077620 0x7f3f48079ae0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f3f500095f0 tx=0x7f3f50005f90 comp rx=0 tx=0).ready entity=mgr.14601 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:27:50.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:50.613+0000 7f3f5cff9640 1 -- 172.21.15.142:0/964846669 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7f3f54086f10 con 0x7f3f60102ae0 2024-09-06T14:27:50.799 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:50.802+0000 7f3f65b64640 1 -- 172.21.15.142:0/964846669 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f3f6010ac50 con 0x7f3f60102ae0 2024-09-06T14:27:50.799 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:50.802+0000 7f3f5cff9640 1 -- 172.21.15.142:0/964846669 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+212 (secure 0 0 0) 0x7f3f5408b930 con 0x7f3f60102ae0 2024-09-06T14:27:50.800 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:27:50.800 INFO:teuthology.orchestra.run.smithi142.stdout:{"status":"HEALTH_WARN","checks":{"MGR_MODULE_DEPENDENCY":{"severity":"HEALTH_WARN","summary":{"message":"Module 'volumes' has failed dependency: No module named 'ceph.fs'","count":1},"muted":false}},"mutes":[]} 2024-09-06T14:27:50.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:50.804+0000 7f3f65b64640 1 -- 172.21.15.142:0/964846669 >> v2:172.21.15.142:6800/2944545612 conn(0x7f3f48077620 msgr2=0x7f3f48079ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:27:50.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:50.804+0000 7f3f65b64640 1 --2- 172.21.15.142:0/964846669 >> v2:172.21.15.142:6800/2944545612 conn(0x7f3f48077620 0x7f3f48079ae0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f3f500095f0 tx=0x7f3f50005f90 comp rx=0 tx=0).stop 2024-09-06T14:27:50.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:50.804+0000 7f3f65b64640 1 -- 172.21.15.142:0/964846669 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3f60102ae0 msgr2=0x7f3f60075200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:27:50.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:50.804+0000 7f3f65b64640 1 --2- 172.21.15.142:0/964846669 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3f60102ae0 0x7f3f60075200 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f3f5402a910 tx=0x7f3f540329c0 comp rx=0 tx=0).stop 2024-09-06T14:27:50.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:50.804+0000 7f3f65b64640 1 -- 172.21.15.142:0/964846669 shutdown_connections 2024-09-06T14:27:50.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:50.804+0000 7f3f65b64640 1 --2- 172.21.15.142:0/964846669 >> v2:172.21.15.142:6800/2944545612 conn(0x7f3f48077620 0x7f3f48079ae0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:50.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:50.804+0000 7f3f65b64640 1 --2- 172.21.15.142:0/964846669 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3f60108e70 0x7f3f60075c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:50.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:50.804+0000 7f3f65b64640 1 --2- 172.21.15.142:0/964846669 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3f601084f0 0x7f3f60075740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:50.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:50.804+0000 7f3f65b64640 1 --2- 172.21.15.142:0/964846669 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3f60102ae0 0x7f3f60075200 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:50.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:50.804+0000 7f3f65b64640 1 -- 172.21.15.142:0/964846669 >> 172.21.15.142:0/964846669 conn(0x7f3f600fe530 msgr2=0x7f3f60100920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:27:50.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:50.804+0000 7f3f65b64640 1 -- 172.21.15.142:0/964846669 shutdown_connections 2024-09-06T14:27:50.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:50.804+0000 7f3f65b64640 1 -- 172.21.15.142:0/964846669 wait complete. 2024-09-06T14:27:51.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:27:50 smithi162 ceph-mon[33787]: pgmap v92: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:27:51.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:27:50 smithi162 ceph-mon[33787]: from='client.? 172.21.15.142:0/964846669' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:27:51.307 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:27:50 smithi142 ceph-mon[37372]: pgmap v92: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:27:51.308 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:27:50 smithi142 ceph-mon[37372]: from='client.? 172.21.15.142:0/964846669' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:27:51.308 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:27:50 smithi142 ceph-mon[29958]: pgmap v92: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:27:51.308 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:27:50 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/964846669' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:27:53.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:27:52 smithi162 ceph-mon[33787]: pgmap v93: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:27:53.459 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:27:52 smithi142 ceph-mon[29958]: pgmap v93: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:27:53.459 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:27:52 smithi142 ceph-mon[37372]: pgmap v93: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:27:54.453 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph health --format=json 2024-09-06T14:27:54.709 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:27:54 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: ::ffff:172.21.15.162 - - [06/Sep/2024:14:27:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-06T14:27:54.715 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:27:55.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:55.222+0000 7f8fb66d9640 1 -- 172.21.15.142:0/2272275647 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8fb01073d0 msgr2=0x7f8fb0107810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:27:55.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:55.222+0000 7f8fb66d9640 1 --2- 172.21.15.142:0/2272275647 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8fb01073d0 0x7f8fb0107810 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f8fa4009900 tx=0x7f8fa402a410 comp rx=0 tx=0).stop 2024-09-06T14:27:55.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:55.222+0000 7f8fb66d9640 1 -- 172.21.15.142:0/2272275647 shutdown_connections 2024-09-06T14:27:55.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:55.222+0000 7f8fb66d9640 1 --2- 172.21.15.142:0/2272275647 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8fb0107d50 0x7f8fb010e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:55.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:55.222+0000 7f8fb66d9640 1 --2- 172.21.15.142:0/2272275647 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8fb01073d0 0x7f8fb0107810 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:55.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:55.222+0000 7f8fb66d9640 1 --2- 172.21.15.142:0/2272275647 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8fb01019c0 0x7f8fb0101da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:55.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:55.222+0000 7f8fb66d9640 1 -- 172.21.15.142:0/2272275647 >> 172.21.15.142:0/2272275647 conn(0x7f8fb00fd430 msgr2=0x7f8fb00ff850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:27:55.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:55.222+0000 7f8fb66d9640 1 -- 172.21.15.142:0/2272275647 shutdown_connections 2024-09-06T14:27:55.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:55.222+0000 7f8fb66d9640 1 -- 172.21.15.142:0/2272275647 wait complete. 2024-09-06T14:27:55.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:55.222+0000 7f8fb66d9640 1 Processor -- start 2024-09-06T14:27:55.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:55.222+0000 7f8fb66d9640 1 -- start start 2024-09-06T14:27:55.232 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:55.223+0000 7f8fb66d9640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8fb01019c0 0x7f8fb019a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:27:55.232 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:55.223+0000 7f8fb66d9640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8fb01073d0 0x7f8fb019ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:27:55.232 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:55.223+0000 7f8fb66d9640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8fb0107d50 0x7f8fb01a01f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:27:55.232 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:55.223+0000 7f8fb66d9640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f8fb0111120 con 0x7f8fb0107d50 2024-09-06T14:27:55.232 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:55.223+0000 7f8fb66d9640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f8fb0110fa0 con 0x7f8fb01073d0 2024-09-06T14:27:55.232 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:55.223+0000 7f8fb66d9640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7f8fb01112a0 con 0x7f8fb01019c0 2024-09-06T14:27:55.232 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:55.223+0000 7f8faffff640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8fb01019c0 0x7f8fb019a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:27:55.232 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:55.223+0000 7f8faffff640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8fb01019c0 0x7f8fb019a910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:36846/0 (socket says 172.21.15.142:36846) 2024-09-06T14:27:55.232 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:55.223+0000 7f8faffff640 1 -- 172.21.15.142:0/3986055958 learned_addr learned my addr 172.21.15.142:0/3986055958 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:27:55.232 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:55.223+0000 7f8fb4c76640 1 --2- 172.21.15.142:0/3986055958 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8fb0107d50 0x7f8fb01a01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:27:55.232 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:55.223+0000 7f8faffff640 1 -- 172.21.15.142:0/3986055958 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8fb01073d0 msgr2=0x7f8fb019ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:27:55.233 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:55.223+0000 7f8faffff640 1 --2- 172.21.15.142:0/3986055958 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8fb01073d0 0x7f8fb019ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:55.233 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:55.223+0000 7f8faffff640 1 -- 172.21.15.142:0/3986055958 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8fb0107d50 msgr2=0x7f8fb01a01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:27:55.233 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:55.223+0000 7f8faffff640 1 --2- 172.21.15.142:0/3986055958 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8fb0107d50 0x7f8fb01a01f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:55.233 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:55.223+0000 7f8faffff640 1 -- 172.21.15.142:0/3986055958 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fb01a09c0 con 0x7f8fb01019c0 2024-09-06T14:27:55.233 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:55.223+0000 7f8faffff640 1 --2- 172.21.15.142:0/3986055958 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8fb01019c0 0x7f8fb019a910 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f8fa000ea60 tx=0x7f8fa000ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:27:55.233 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:55.223+0000 7f8fad7fa640 1 -- 172.21.15.142:0/3986055958 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fa000cde0 con 0x7f8fb01019c0 2024-09-06T14:27:55.233 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:55.224+0000 7f8fb66d9640 1 -- 172.21.15.142:0/3986055958 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fb01a0cb0 con 0x7f8fb01019c0 2024-09-06T14:27:55.233 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:55.224+0000 7f8fad7fa640 1 -- 172.21.15.142:0/3986055958 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7f8fa00035f0 con 0x7f8fb01019c0 2024-09-06T14:27:55.233 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:55.224+0000 7f8fad7fa640 1 -- 172.21.15.142:0/3986055958 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fa00044a0 con 0x7f8fb01019c0 2024-09-06T14:27:55.233 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:55.224+0000 7f8fb66d9640 1 -- 172.21.15.142:0/3986055958 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fb01a7540 con 0x7f8fb01019c0 2024-09-06T14:27:55.233 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:55.225+0000 7f8fb66d9640 1 -- 172.21.15.142:0/3986055958 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8fb0101da0 con 0x7f8fb01019c0 2024-09-06T14:27:55.233 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:55.225+0000 7f8fad7fa640 1 -- 172.21.15.142:0/3986055958 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 22) ==== 99844+0+0 (secure 0 0 0) 0x7f8fa0004000 con 0x7f8fb01019c0 2024-09-06T14:27:55.233 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:55.226+0000 7f8fad7fa640 1 --2- 172.21.15.142:0/3986055958 >> v2:172.21.15.142:6800/2944545612 conn(0x7f8f98077620 0x7f8f98079ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:27:55.234 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:55.226+0000 7f8faf7fe640 1 --2- 172.21.15.142:0/3986055958 >> v2:172.21.15.142:6800/2944545612 conn(0x7f8f98077620 0x7f8f98079ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:27:55.234 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:55.226+0000 7f8fad7fa640 1 -- 172.21.15.142:0/3986055958 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(64..64 src has 1..64) ==== 6130+0+0 (secure 0 0 0) 0x7f8fa0099d80 con 0x7f8fb01019c0 2024-09-06T14:27:55.234 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:55.226+0000 7f8faf7fe640 1 --2- 172.21.15.142:0/3986055958 >> v2:172.21.15.142:6800/2944545612 conn(0x7f8f98077620 0x7f8f98079ae0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f8fa402a930 tx=0x7f8fa4031400 comp rx=0 tx=0).ready entity=mgr.14601 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:27:55.234 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:55.229+0000 7f8fad7fa640 1 -- 172.21.15.142:0/3986055958 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7f8fa0062e40 con 0x7f8fb01019c0 2024-09-06T14:27:55.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:27:54 smithi162 ceph-mon[33787]: pgmap v94: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:27:55.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:55.417+0000 7f8fb66d9640 1 -- 172.21.15.142:0/3986055958 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f8fb01a7710 con 0x7f8fb01019c0 2024-09-06T14:27:55.414 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:27:54 smithi142 ceph-mon[29958]: pgmap v94: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:27:55.415 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:27:54 smithi142 ceph-mon[37372]: pgmap v94: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:27:55.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:55.417+0000 7f8fad7fa640 1 -- 172.21.15.142:0/3986055958 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+212 (secure 0 0 0) 0x7f8fa0083e20 con 0x7f8fb01019c0 2024-09-06T14:27:55.415 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:27:55.415 INFO:teuthology.orchestra.run.smithi142.stdout:{"status":"HEALTH_WARN","checks":{"MGR_MODULE_DEPENDENCY":{"severity":"HEALTH_WARN","summary":{"message":"Module 'volumes' has failed dependency: No module named 'ceph.fs'","count":1},"muted":false}},"mutes":[]} 2024-09-06T14:27:55.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:55.419+0000 7f8fb66d9640 1 -- 172.21.15.142:0/3986055958 >> v2:172.21.15.142:6800/2944545612 conn(0x7f8f98077620 msgr2=0x7f8f98079ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:27:55.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:55.419+0000 7f8fb66d9640 1 --2- 172.21.15.142:0/3986055958 >> v2:172.21.15.142:6800/2944545612 conn(0x7f8f98077620 0x7f8f98079ae0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f8fa402a930 tx=0x7f8fa4031400 comp rx=0 tx=0).stop 2024-09-06T14:27:55.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:55.419+0000 7f8fb66d9640 1 -- 172.21.15.142:0/3986055958 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8fb01019c0 msgr2=0x7f8fb019a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:27:55.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:55.419+0000 7f8fb66d9640 1 --2- 172.21.15.142:0/3986055958 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8fb01019c0 0x7f8fb019a910 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f8fa000ea60 tx=0x7f8fa000ef20 comp rx=0 tx=0).stop 2024-09-06T14:27:55.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:55.419+0000 7f8fb66d9640 1 -- 172.21.15.142:0/3986055958 shutdown_connections 2024-09-06T14:27:55.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:55.419+0000 7f8fb66d9640 1 --2- 172.21.15.142:0/3986055958 >> v2:172.21.15.142:6800/2944545612 conn(0x7f8f98077620 0x7f8f98079ae0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:55.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:55.419+0000 7f8fb66d9640 1 --2- 172.21.15.142:0/3986055958 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8fb0107d50 0x7f8fb01a01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:55.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:55.419+0000 7f8fb66d9640 1 --2- 172.21.15.142:0/3986055958 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8fb01073d0 0x7f8fb019ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:55.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:55.419+0000 7f8fb66d9640 1 --2- 172.21.15.142:0/3986055958 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8fb01019c0 0x7f8fb019a910 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:55.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:55.419+0000 7f8fb66d9640 1 -- 172.21.15.142:0/3986055958 >> 172.21.15.142:0/3986055958 conn(0x7f8fb00fd430 msgr2=0x7f8fb0100820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:27:55.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:55.420+0000 7f8fb66d9640 1 -- 172.21.15.142:0/3986055958 shutdown_connections 2024-09-06T14:27:55.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:55.420+0000 7f8fb66d9640 1 -- 172.21.15.142:0/3986055958 wait complete. 2024-09-06T14:27:56.110 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:27:55 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/3986055958' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:27:56.111 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:27:55 smithi142 ceph-mon[29958]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-06T14:27:56.111 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:27:55 smithi142 ceph-mon[37372]: from='client.? 172.21.15.142:0/3986055958' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:27:56.111 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:27:55 smithi142 ceph-mon[37372]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-06T14:27:56.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:27:55 smithi162 ceph-mon[33787]: from='client.? 172.21.15.142:0/3986055958' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:27:56.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:27:55 smithi162 ceph-mon[33787]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-06T14:27:57.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:27:56 smithi162 ceph-mon[33787]: pgmap v95: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:27:57.298 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 06 14:27:57 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-iscsi-iscsi-a[59853]: debug there is no tcmu-runner data available 2024-09-06T14:27:57.459 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:27:56 smithi142 ceph-mon[29958]: pgmap v95: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:27:57.459 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:27:56 smithi142 ceph-mon[37372]: pgmap v95: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:27:59.141 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph health --format=json 2024-09-06T14:27:59.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:27:58 smithi162 ceph-mon[33787]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-06T14:27:59.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:27:58 smithi162 ceph-mon[33787]: pgmap v96: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:27:59.402 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:27:59.459 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:27:58 smithi142 ceph-mon[29958]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-06T14:27:59.459 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:27:58 smithi142 ceph-mon[29958]: pgmap v96: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:27:59.459 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:27:58 smithi142 ceph-mon[37372]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-06T14:27:59.460 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:27:58 smithi142 ceph-mon[37372]: pgmap v96: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:27:59.872 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:59.869+0000 7f6c009fd640 1 -- 172.21.15.142:0/630800409 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6bfc1073f0 msgr2=0x7f6bfc107830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:27:59.872 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:59.869+0000 7f6c009fd640 1 --2- 172.21.15.142:0/630800409 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6bfc1073f0 0x7f6bfc107830 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f6bf0009900 tx=0x7f6bf002a3f0 comp rx=0 tx=0).stop 2024-09-06T14:27:59.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:59.870+0000 7f6c009fd640 1 -- 172.21.15.142:0/630800409 shutdown_connections 2024-09-06T14:27:59.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:59.870+0000 7f6c009fd640 1 --2- 172.21.15.142:0/630800409 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6bfc107d70 0x7f6bfc10e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:59.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:59.870+0000 7f6c009fd640 1 --2- 172.21.15.142:0/630800409 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6bfc1073f0 0x7f6bfc107830 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:59.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:59.870+0000 7f6c009fd640 1 --2- 172.21.15.142:0/630800409 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6bfc1019e0 0x7f6bfc101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:59.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:59.870+0000 7f6c009fd640 1 -- 172.21.15.142:0/630800409 >> 172.21.15.142:0/630800409 conn(0x7f6bfc0fd450 msgr2=0x7f6bfc0ff870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:27:59.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:59.870+0000 7f6c009fd640 1 -- 172.21.15.142:0/630800409 shutdown_connections 2024-09-06T14:27:59.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:59.870+0000 7f6c009fd640 1 -- 172.21.15.142:0/630800409 wait complete. 2024-09-06T14:27:59.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:59.870+0000 7f6c009fd640 1 Processor -- start 2024-09-06T14:27:59.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:59.870+0000 7f6c009fd640 1 -- start start 2024-09-06T14:27:59.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:59.871+0000 7f6c009fd640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6bfc1019e0 0x7f6bfc10f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:27:59.874 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:59.871+0000 7f6c009fd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6bfc1073f0 0x7f6bfc10f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:27:59.874 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:59.871+0000 7f6c009fd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6bfc107d70 0x7f6bfc10fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:27:59.874 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:59.871+0000 7f6c009fd640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f6bfc112240 con 0x7f6bfc107d70 2024-09-06T14:27:59.874 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:59.871+0000 7f6c009fd640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6bfc1120c0 con 0x7f6bfc1073f0 2024-09-06T14:27:59.874 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:59.871+0000 7f6c009fd640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7f6bfc1123c0 con 0x7f6bfc1019e0 2024-09-06T14:27:59.874 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:59.871+0000 7f6bfa59c640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6bfc1019e0 0x7f6bfc10f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:27:59.874 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:59.871+0000 7f6bfa59c640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6bfc1019e0 0x7f6bfc10f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:60370/0 (socket says 172.21.15.142:60370) 2024-09-06T14:27:59.874 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:59.871+0000 7f6bfa59c640 1 -- 172.21.15.142:0/1657842553 learned_addr learned my addr 172.21.15.142:0/1657842553 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:27:59.876 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:59.871+0000 7f6bfad9d640 1 --2- 172.21.15.142:0/1657842553 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6bfc107d70 0x7f6bfc10fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:27:59.876 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:59.871+0000 7f6bf9d9b640 1 --2- 172.21.15.142:0/1657842553 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6bfc1073f0 0x7f6bfc10f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:27:59.876 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:59.871+0000 7f6bfad9d640 1 -- 172.21.15.142:0/1657842553 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6bfc1019e0 msgr2=0x7f6bfc10f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:27:59.876 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:59.871+0000 7f6bfad9d640 1 --2- 172.21.15.142:0/1657842553 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6bfc1019e0 0x7f6bfc10f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:59.876 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:59.871+0000 7f6bfad9d640 1 -- 172.21.15.142:0/1657842553 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6bfc1073f0 msgr2=0x7f6bfc10f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:27:59.876 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:59.871+0000 7f6bfad9d640 1 --2- 172.21.15.142:0/1657842553 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6bfc1073f0 0x7f6bfc10f5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:27:59.876 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:59.871+0000 7f6bfad9d640 1 -- 172.21.15.142:0/1657842553 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bfc1a6ad0 con 0x7f6bfc107d70 2024-09-06T14:27:59.876 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:59.871+0000 7f6bfa59c640 1 --2- 172.21.15.142:0/1657842553 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6bfc1019e0 0x7f6bfc10f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-06T14:27:59.876 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:59.871+0000 7f6bfad9d640 1 --2- 172.21.15.142:0/1657842553 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6bfc107d70 0x7f6bfc10fb00 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f6bf400ca30 tx=0x7f6bf400cef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:27:59.876 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:59.871+0000 7f6beb7fe640 1 -- 172.21.15.142:0/1657842553 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bf4007d90 con 0x7f6bfc107d70 2024-09-06T14:27:59.876 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:59.871+0000 7f6beb7fe640 1 -- 172.21.15.142:0/1657842553 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7f6bf4005020 con 0x7f6bfc107d70 2024-09-06T14:27:59.876 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:59.871+0000 7f6c009fd640 1 -- 172.21.15.142:0/1657842553 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6bfc1a6dc0 con 0x7f6bfc107d70 2024-09-06T14:27:59.877 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:59.871+0000 7f6beb7fe640 1 -- 172.21.15.142:0/1657842553 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bf400f6b0 con 0x7f6bfc107d70 2024-09-06T14:27:59.877 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:59.872+0000 7f6c009fd640 1 -- 172.21.15.142:0/1657842553 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6bfc1a7220 con 0x7f6bfc107d70 2024-09-06T14:27:59.877 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:59.873+0000 7f6c009fd640 1 -- 172.21.15.142:0/1657842553 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6bfc101dc0 con 0x7f6bfc107d70 2024-09-06T14:27:59.877 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:59.873+0000 7f6beb7fe640 1 -- 172.21.15.142:0/1657842553 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 22) ==== 99844+0+0 (secure 0 0 0) 0x7f6bf4004b20 con 0x7f6bfc107d70 2024-09-06T14:27:59.877 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:59.873+0000 7f6beb7fe640 1 --2- 172.21.15.142:0/1657842553 >> v2:172.21.15.142:6800/2944545612 conn(0x7f6be4077620 0x7f6be4079ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:27:59.877 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:59.874+0000 7f6bfa59c640 1 --2- 172.21.15.142:0/1657842553 >> v2:172.21.15.142:6800/2944545612 conn(0x7f6be4077620 0x7f6be4079ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:27:59.877 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:59.874+0000 7f6beb7fe640 1 -- 172.21.15.142:0/1657842553 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 6130+0+0 (secure 0 0 0) 0x7f6bf4099ba0 con 0x7f6bfc107d70 2024-09-06T14:27:59.877 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:59.875+0000 7f6bfa59c640 1 --2- 172.21.15.142:0/1657842553 >> v2:172.21.15.142:6800/2944545612 conn(0x7f6be4077620 0x7f6be4079ae0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f6bec0095f0 tx=0x7f6bec005f90 comp rx=0 tx=0).ready entity=mgr.14601 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:27:59.877 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:27:59.878+0000 7f6beb7fe640 1 -- 172.21.15.142:0/1657842553 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7f6bf4014070 con 0x7f6bfc107d70 2024-09-06T14:28:00.066 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:00.069+0000 7f6c009fd640 1 -- 172.21.15.142:0/1657842553 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f6bfc109b80 con 0x7f6bfc107d70 2024-09-06T14:28:00.066 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:00.069+0000 7f6beb7fe640 1 -- 172.21.15.142:0/1657842553 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+212 (secure 0 0 0) 0x7f6bf4066dd0 con 0x7f6bfc107d70 2024-09-06T14:28:00.067 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:28:00.067 INFO:teuthology.orchestra.run.smithi142.stdout:{"status":"HEALTH_WARN","checks":{"MGR_MODULE_DEPENDENCY":{"severity":"HEALTH_WARN","summary":{"message":"Module 'volumes' has failed dependency: No module named 'ceph.fs'","count":1},"muted":false}},"mutes":[]} 2024-09-06T14:28:00.068 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:00.071+0000 7f6c009fd640 1 -- 172.21.15.142:0/1657842553 >> v2:172.21.15.142:6800/2944545612 conn(0x7f6be4077620 msgr2=0x7f6be4079ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:28:00.069 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:00.071+0000 7f6c009fd640 1 --2- 172.21.15.142:0/1657842553 >> v2:172.21.15.142:6800/2944545612 conn(0x7f6be4077620 0x7f6be4079ae0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f6bec0095f0 tx=0x7f6bec005f90 comp rx=0 tx=0).stop 2024-09-06T14:28:00.069 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:00.071+0000 7f6c009fd640 1 -- 172.21.15.142:0/1657842553 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6bfc107d70 msgr2=0x7f6bfc10fb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:28:00.069 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:00.071+0000 7f6c009fd640 1 --2- 172.21.15.142:0/1657842553 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6bfc107d70 0x7f6bfc10fb00 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f6bf400ca30 tx=0x7f6bf400cef0 comp rx=0 tx=0).stop 2024-09-06T14:28:00.069 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:00.071+0000 7f6c009fd640 1 -- 172.21.15.142:0/1657842553 shutdown_connections 2024-09-06T14:28:00.069 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:00.071+0000 7f6c009fd640 1 --2- 172.21.15.142:0/1657842553 >> v2:172.21.15.142:6800/2944545612 conn(0x7f6be4077620 0x7f6be4079ae0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:00.069 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:00.071+0000 7f6c009fd640 1 --2- 172.21.15.142:0/1657842553 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6bfc107d70 0x7f6bfc10fb00 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:00.069 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:00.071+0000 7f6c009fd640 1 --2- 172.21.15.142:0/1657842553 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6bfc1073f0 0x7f6bfc10f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:00.069 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:00.071+0000 7f6c009fd640 1 --2- 172.21.15.142:0/1657842553 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6bfc1019e0 0x7f6bfc10f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:00.069 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:00.071+0000 7f6c009fd640 1 -- 172.21.15.142:0/1657842553 >> 172.21.15.142:0/1657842553 conn(0x7f6bfc0fd450 msgr2=0x7f6bfc0ff840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:28:00.069 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:00.071+0000 7f6c009fd640 1 -- 172.21.15.142:0/1657842553 shutdown_connections 2024-09-06T14:28:00.069 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:00.071+0000 7f6c009fd640 1 -- 172.21.15.142:0/1657842553 wait complete. 2024-09-06T14:28:01.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:28:00 smithi162 ceph-mon[33787]: pgmap v97: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:28:01.298 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:28:00 smithi162 ceph-mon[33787]: from='client.? 172.21.15.142:0/1657842553' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:28:01.459 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:28:00 smithi142 ceph-mon[29958]: pgmap v97: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:28:01.459 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:28:00 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/1657842553' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:28:01.459 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:28:00 smithi142 ceph-mon[37372]: pgmap v97: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:28:01.459 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:28:00 smithi142 ceph-mon[37372]: from='client.? 172.21.15.142:0/1657842553' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:28:03.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:28:03 smithi162 ceph-mon[33787]: pgmap v98: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:28:03.459 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:28:03 smithi142 ceph-mon[29958]: pgmap v98: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:28:03.459 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:28:03 smithi142 ceph-mon[37372]: pgmap v98: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:28:03.669 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph health --format=json 2024-09-06T14:28:03.929 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:28:04.396 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:04.367+0000 7f016b922640 1 -- 172.21.15.142:0/4191692783 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f01641017c0 msgr2=0x7f0164101ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:28:04.396 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:04.367+0000 7f016b922640 1 --2- 172.21.15.142:0/4191692783 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f01641017c0 0x7f0164101ba0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f0160009900 tx=0x7f016002a410 comp rx=0 tx=0).stop 2024-09-06T14:28:04.396 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:04.367+0000 7f016b922640 1 -- 172.21.15.142:0/4191692783 shutdown_connections 2024-09-06T14:28:04.396 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:04.367+0000 7f016b922640 1 --2- 172.21.15.142:0/4191692783 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0164106c00 0x7f0164107050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:04.396 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:04.367+0000 7f016b922640 1 --2- 172.21.15.142:0/4191692783 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f01641020e0 0x7f01641065b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:04.396 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:04.367+0000 7f016b922640 1 --2- 172.21.15.142:0/4191692783 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f01641017c0 0x7f0164101ba0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:04.397 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:04.367+0000 7f016b922640 1 -- 172.21.15.142:0/4191692783 >> 172.21.15.142:0/4191692783 conn(0x7f01640fd430 msgr2=0x7f01640ff850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:28:04.397 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:04.367+0000 7f016b922640 1 -- 172.21.15.142:0/4191692783 shutdown_connections 2024-09-06T14:28:04.397 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:04.367+0000 7f016b922640 1 -- 172.21.15.142:0/4191692783 wait complete. 2024-09-06T14:28:04.397 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:04.368+0000 7f016b922640 1 Processor -- start 2024-09-06T14:28:04.397 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:04.368+0000 7f016b922640 1 -- start start 2024-09-06T14:28:04.397 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:04.368+0000 7f016b922640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f01641017c0 0x7f016419ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:28:04.397 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:04.368+0000 7f016b922640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f01641020e0 0x7f016419f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:28:04.397 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:04.368+0000 7f016b922640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0164106c00 0x7f01641a4640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:28:04.397 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:04.368+0000 7f016b922640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f01641187d0 con 0x7f01641020e0 2024-09-06T14:28:04.397 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:04.368+0000 7f016b922640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f0164118650 con 0x7f0164106c00 2024-09-06T14:28:04.397 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:04.368+0000 7f016b922640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7f0164118950 con 0x7f01641017c0 2024-09-06T14:28:04.397 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:04.368+0000 7f0168ebd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f01641020e0 0x7f016419f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:28:04.397 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:04.368+0000 7f01696be640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f01641017c0 0x7f016419ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:28:04.397 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:04.368+0000 7f0168ebd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f01641020e0 0x7f016419f2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:38232/0 (socket says 172.21.15.142:38232) 2024-09-06T14:28:04.398 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:04.368+0000 7f0168ebd640 1 -- 172.21.15.142:0/1137418913 learned_addr learned my addr 172.21.15.142:0/1137418913 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:28:04.398 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:04.368+0000 7f0169ebf640 1 --2- 172.21.15.142:0/1137418913 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0164106c00 0x7f01641a4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:28:04.398 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:04.369+0000 7f0168ebd640 1 -- 172.21.15.142:0/1137418913 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f01641017c0 msgr2=0x7f016419ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:28:04.398 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:04.369+0000 7f0168ebd640 1 --2- 172.21.15.142:0/1137418913 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f01641017c0 0x7f016419ed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:04.398 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:04.369+0000 7f0168ebd640 1 -- 172.21.15.142:0/1137418913 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0164106c00 msgr2=0x7f01641a4640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:28:04.398 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:04.369+0000 7f0168ebd640 1 --2- 172.21.15.142:0/1137418913 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0164106c00 0x7f01641a4640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:04.398 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:04.369+0000 7f0168ebd640 1 -- 172.21.15.142:0/1137418913 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01641a4e10 con 0x7f01641020e0 2024-09-06T14:28:04.398 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:04.369+0000 7f01696be640 1 --2- 172.21.15.142:0/1137418913 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f01641017c0 0x7f016419ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-06T14:28:04.398 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:04.369+0000 7f0168ebd640 1 --2- 172.21.15.142:0/1137418913 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f01641020e0 0x7f016419f2a0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f015400c970 tx=0x7f015400cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:28:04.398 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:04.369+0000 7f015a7fc640 1 -- 172.21.15.142:0/1137418913 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f015400ae60 con 0x7f01641020e0 2024-09-06T14:28:04.398 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:04.369+0000 7f015a7fc640 1 -- 172.21.15.142:0/1137418913 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7f0154007e30 con 0x7f01641020e0 2024-09-06T14:28:04.398 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:04.369+0000 7f016b922640 1 -- 172.21.15.142:0/1137418913 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01641a5100 con 0x7f01641020e0 2024-09-06T14:28:04.398 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:04.369+0000 7f015a7fc640 1 -- 172.21.15.142:0/1137418913 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0154003ca0 con 0x7f01641020e0 2024-09-06T14:28:04.398 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:04.369+0000 7f016b922640 1 -- 172.21.15.142:0/1137418913 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01641ab7c0 con 0x7f01641020e0 2024-09-06T14:28:04.398 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:04.370+0000 7f016b922640 1 -- 172.21.15.142:0/1137418913 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01641065b0 con 0x7f01641020e0 2024-09-06T14:28:04.399 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:04.371+0000 7f015a7fc640 1 -- 172.21.15.142:0/1137418913 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 22) ==== 99844+0+0 (secure 0 0 0) 0x7f01540079b0 con 0x7f01641020e0 2024-09-06T14:28:04.399 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:04.371+0000 7f015a7fc640 1 --2- 172.21.15.142:0/1137418913 >> v2:172.21.15.142:6800/2944545612 conn(0x7f0150077620 0x7f0150079ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:28:04.399 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:04.371+0000 7f01696be640 1 --2- 172.21.15.142:0/1137418913 >> v2:172.21.15.142:6800/2944545612 conn(0x7f0150077620 0x7f0150079ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:28:04.399 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:04.371+0000 7f015a7fc640 1 -- 172.21.15.142:0/1137418913 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 6130+0+0 (secure 0 0 0) 0x7f0154099490 con 0x7f01641020e0 2024-09-06T14:28:04.399 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:04.372+0000 7f01696be640 1 --2- 172.21.15.142:0/1137418913 >> v2:172.21.15.142:6800/2944545612 conn(0x7f0150077620 0x7f0150079ae0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f0160031440 tx=0x7f0160031850 comp rx=0 tx=0).ready entity=mgr.14601 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:28:04.399 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:04.375+0000 7f015a7fc640 1 -- 172.21.15.142:0/1137418913 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7f0154014030 con 0x7f01641020e0 2024-09-06T14:28:04.563 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:04.566+0000 7f016b922640 1 -- 172.21.15.142:0/1137418913 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f01641aba80 con 0x7f01641020e0 2024-09-06T14:28:04.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:04.566+0000 7f015a7fc640 1 -- 172.21.15.142:0/1137418913 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+212 (secure 0 0 0) 0x7f0154003e40 con 0x7f01641020e0 2024-09-06T14:28:04.564 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:28:04.564 INFO:teuthology.orchestra.run.smithi142.stdout:{"status":"HEALTH_WARN","checks":{"MGR_MODULE_DEPENDENCY":{"severity":"HEALTH_WARN","summary":{"message":"Module 'volumes' has failed dependency: No module named 'ceph.fs'","count":1},"muted":false}},"mutes":[]} 2024-09-06T14:28:04.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:04.568+0000 7f016b922640 1 -- 172.21.15.142:0/1137418913 >> v2:172.21.15.142:6800/2944545612 conn(0x7f0150077620 msgr2=0x7f0150079ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:28:04.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:04.568+0000 7f016b922640 1 --2- 172.21.15.142:0/1137418913 >> v2:172.21.15.142:6800/2944545612 conn(0x7f0150077620 0x7f0150079ae0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f0160031440 tx=0x7f0160031850 comp rx=0 tx=0).stop 2024-09-06T14:28:04.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:04.568+0000 7f016b922640 1 -- 172.21.15.142:0/1137418913 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f01641020e0 msgr2=0x7f016419f2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:28:04.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:04.568+0000 7f016b922640 1 --2- 172.21.15.142:0/1137418913 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f01641020e0 0x7f016419f2a0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f015400c970 tx=0x7f015400cd70 comp rx=0 tx=0).stop 2024-09-06T14:28:04.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:04.568+0000 7f016b922640 1 -- 172.21.15.142:0/1137418913 shutdown_connections 2024-09-06T14:28:04.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:04.568+0000 7f016b922640 1 --2- 172.21.15.142:0/1137418913 >> v2:172.21.15.142:6800/2944545612 conn(0x7f0150077620 0x7f0150079ae0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:04.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:04.568+0000 7f016b922640 1 --2- 172.21.15.142:0/1137418913 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0164106c00 0x7f01641a4640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:04.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:04.568+0000 7f016b922640 1 --2- 172.21.15.142:0/1137418913 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f01641020e0 0x7f016419f2a0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:04.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:04.568+0000 7f016b922640 1 --2- 172.21.15.142:0/1137418913 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f01641017c0 0x7f016419ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:04.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:04.568+0000 7f016b922640 1 -- 172.21.15.142:0/1137418913 >> 172.21.15.142:0/1137418913 conn(0x7f01640fd430 msgr2=0x7f0164104530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:28:04.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:04.568+0000 7f016b922640 1 -- 172.21.15.142:0/1137418913 shutdown_connections 2024-09-06T14:28:04.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:04.568+0000 7f016b922640 1 -- 172.21.15.142:0/1137418913 wait complete. 2024-09-06T14:28:04.709 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:28:04 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: ::ffff:172.21.15.162 - - [06/Sep/2024:14:28:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-06T14:28:05.257 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:28:05 smithi142 ceph-mon[29958]: pgmap v99: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:28:05.257 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:28:05 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/1137418913' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:28:05.257 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:28:05 smithi142 ceph-mon[37372]: pgmap v99: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:28:05.257 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:28:05 smithi142 ceph-mon[37372]: from='client.? 172.21.15.142:0/1137418913' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:28:05.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:28:05 smithi162 ceph-mon[33787]: pgmap v99: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:28:05.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:28:05 smithi162 ceph-mon[33787]: from='client.? 172.21.15.142:0/1137418913' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:28:07.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:28:07 smithi162 ceph-mon[33787]: pgmap v100: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:28:07.298 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 06 14:28:07 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-iscsi-iscsi-a[59853]: debug there is no tcmu-runner data available 2024-09-06T14:28:07.459 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:28:07 smithi142 ceph-mon[29958]: pgmap v100: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:28:07.459 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:28:07 smithi142 ceph-mon[37372]: pgmap v100: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:28:08.260 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph health --format=json 2024-09-06T14:28:08.523 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:28:09.028 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:09.001+0000 7f70977f6640 1 -- 172.21.15.142:0/1333347044 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7090107d50 msgr2=0x7f709010e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:28:09.028 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:09.001+0000 7f70977f6640 1 --2- 172.21.15.142:0/1333347044 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7090107d50 0x7f709010e550 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f7088009be0 tx=0x7f708802a6c0 comp rx=0 tx=0).stop 2024-09-06T14:28:09.028 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:09.001+0000 7f70977f6640 1 -- 172.21.15.142:0/1333347044 shutdown_connections 2024-09-06T14:28:09.028 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:09.001+0000 7f70977f6640 1 --2- 172.21.15.142:0/1333347044 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7090107d50 0x7f709010e550 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:09.028 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:09.001+0000 7f70977f6640 1 --2- 172.21.15.142:0/1333347044 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f70901073d0 0x7f7090107810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:09.028 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:09.001+0000 7f70977f6640 1 --2- 172.21.15.142:0/1333347044 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f70901019c0 0x7f7090101da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:09.028 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:09.001+0000 7f70977f6640 1 -- 172.21.15.142:0/1333347044 >> 172.21.15.142:0/1333347044 conn(0x7f70900fd430 msgr2=0x7f70900ff850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:28:09.028 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:09.001+0000 7f70977f6640 1 -- 172.21.15.142:0/1333347044 shutdown_connections 2024-09-06T14:28:09.028 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:09.001+0000 7f70977f6640 1 -- 172.21.15.142:0/1333347044 wait complete. 2024-09-06T14:28:09.028 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:09.002+0000 7f70977f6640 1 Processor -- start 2024-09-06T14:28:09.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:09.002+0000 7f70977f6640 1 -- start start 2024-09-06T14:28:09.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:09.002+0000 7f70977f6640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f70901019c0 0x7f7090199490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:28:09.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:09.002+0000 7f70977f6640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f70901073d0 0x7f70901999d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:28:09.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:09.002+0000 7f70977f6640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7090107d50 0x7f7090199f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:28:09.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:09.002+0000 7f70977f6640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f7090111120 con 0x7f70901019c0 2024-09-06T14:28:09.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:09.002+0000 7f70977f6640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7090110fa0 con 0x7f70901073d0 2024-09-06T14:28:09.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:09.002+0000 7f70977f6640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7f70901112a0 con 0x7f7090107d50 2024-09-06T14:28:09.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:09.002+0000 7f7095592640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f70901019c0 0x7f7090199490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:28:09.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:09.002+0000 7f7095592640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f70901019c0 0x7f7090199490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:46348/0 (socket says 172.21.15.142:46348) 2024-09-06T14:28:09.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:09.002+0000 7f7095592640 1 -- 172.21.15.142:0/2723448437 learned_addr learned my addr 172.21.15.142:0/2723448437 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:28:09.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:09.002+0000 7f7095d93640 1 --2- 172.21.15.142:0/2723448437 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7090107d50 0x7f7090199f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:28:09.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:09.002+0000 7f7094d91640 1 --2- 172.21.15.142:0/2723448437 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f70901073d0 0x7f70901999d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:28:09.030 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:09.002+0000 7f7095592640 1 -- 172.21.15.142:0/2723448437 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7090107d50 msgr2=0x7f7090199f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:28:09.030 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:09.002+0000 7f7095592640 1 --2- 172.21.15.142:0/2723448437 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7090107d50 0x7f7090199f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:09.030 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:09.002+0000 7f7095592640 1 -- 172.21.15.142:0/2723448437 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f70901073d0 msgr2=0x7f70901999d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:28:09.030 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:09.002+0000 7f7095592640 1 --2- 172.21.15.142:0/2723448437 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f70901073d0 0x7f70901999d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:09.030 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:09.002+0000 7f7095592640 1 -- 172.21.15.142:0/2723448437 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70901a4870 con 0x7f70901019c0 2024-09-06T14:28:09.030 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:09.002+0000 7f7095592640 1 --2- 172.21.15.142:0/2723448437 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f70901019c0 0x7f7090199490 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f708c00e9f0 tx=0x7f708c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:28:09.030 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:09.003+0000 7f70867fc640 1 -- 172.21.15.142:0/2723448437 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f708c00cd80 con 0x7f70901019c0 2024-09-06T14:28:09.030 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:09.003+0000 7f70867fc640 1 -- 172.21.15.142:0/2723448437 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7f708c0108b0 con 0x7f70901019c0 2024-09-06T14:28:09.030 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:09.003+0000 7f70977f6640 1 -- 172.21.15.142:0/2723448437 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70901972e0 con 0x7f70901019c0 2024-09-06T14:28:09.030 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:09.003+0000 7f70867fc640 1 -- 172.21.15.142:0/2723448437 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f708c018870 con 0x7f70901019c0 2024-09-06T14:28:09.030 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:09.003+0000 7f70977f6640 1 -- 172.21.15.142:0/2723448437 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7090197840 con 0x7f70901019c0 2024-09-06T14:28:09.030 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:09.004+0000 7f70977f6640 1 -- 172.21.15.142:0/2723448437 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7090101da0 con 0x7f70901019c0 2024-09-06T14:28:09.030 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:09.004+0000 7f70867fc640 1 -- 172.21.15.142:0/2723448437 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 22) ==== 99844+0+0 (secure 0 0 0) 0x7f708c010430 con 0x7f70901019c0 2024-09-06T14:28:09.030 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:09.005+0000 7f70867fc640 1 --2- 172.21.15.142:0/2723448437 >> v2:172.21.15.142:6800/2944545612 conn(0x7f707c077620 0x7f707c079ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:28:09.031 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:09.005+0000 7f7094d91640 1 --2- 172.21.15.142:0/2723448437 >> v2:172.21.15.142:6800/2944545612 conn(0x7f707c077620 0x7f707c079ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:28:09.031 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:09.005+0000 7f70867fc640 1 -- 172.21.15.142:0/2723448437 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 6130+0+0 (secure 0 0 0) 0x7f708c099ce0 con 0x7f70901019c0 2024-09-06T14:28:09.031 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:09.005+0000 7f7094d91640 1 --2- 172.21.15.142:0/2723448437 >> v2:172.21.15.142:6800/2944545612 conn(0x7f707c077620 0x7f707c079ae0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f709006b3e0 tx=0x7f7080012040 comp rx=0 tx=0).ready entity=mgr.14601 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:28:09.031 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:09.009+0000 7f70867fc640 1 -- 172.21.15.142:0/2723448437 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7f708c066fb0 con 0x7f70901019c0 2024-09-06T14:28:09.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:09.197+0000 7f70977f6640 1 -- 172.21.15.142:0/2723448437 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f7090109b30 con 0x7f70901019c0 2024-09-06T14:28:09.195 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:28:09 smithi142 ceph-mon[37372]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-06T14:28:09.195 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:28:09 smithi142 ceph-mon[37372]: pgmap v101: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:28:09.195 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:28:09 smithi142 ceph-mon[29958]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-06T14:28:09.196 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:28:09 smithi142 ceph-mon[29958]: pgmap v101: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:28:09.196 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:09.198+0000 7f70867fc640 1 -- 172.21.15.142:0/2723448437 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+212 (secure 0 0 0) 0x7f708c06b9d0 con 0x7f70901019c0 2024-09-06T14:28:09.196 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:28:09.196 INFO:teuthology.orchestra.run.smithi142.stdout:{"status":"HEALTH_WARN","checks":{"MGR_MODULE_DEPENDENCY":{"severity":"HEALTH_WARN","summary":{"message":"Module 'volumes' has failed dependency: No module named 'ceph.fs'","count":1},"muted":false}},"mutes":[]} 2024-09-06T14:28:09.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:09.200+0000 7f70977f6640 1 -- 172.21.15.142:0/2723448437 >> v2:172.21.15.142:6800/2944545612 conn(0x7f707c077620 msgr2=0x7f707c079ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:28:09.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:09.200+0000 7f70977f6640 1 --2- 172.21.15.142:0/2723448437 >> v2:172.21.15.142:6800/2944545612 conn(0x7f707c077620 0x7f707c079ae0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f709006b3e0 tx=0x7f7080012040 comp rx=0 tx=0).stop 2024-09-06T14:28:09.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:09.200+0000 7f70977f6640 1 -- 172.21.15.142:0/2723448437 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f70901019c0 msgr2=0x7f7090199490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:28:09.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:09.200+0000 7f70977f6640 1 --2- 172.21.15.142:0/2723448437 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f70901019c0 0x7f7090199490 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f708c00e9f0 tx=0x7f708c00eeb0 comp rx=0 tx=0).stop 2024-09-06T14:28:09.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:09.200+0000 7f70977f6640 1 -- 172.21.15.142:0/2723448437 shutdown_connections 2024-09-06T14:28:09.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:09.200+0000 7f70977f6640 1 --2- 172.21.15.142:0/2723448437 >> v2:172.21.15.142:6800/2944545612 conn(0x7f707c077620 0x7f707c079ae0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:09.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:09.200+0000 7f70977f6640 1 --2- 172.21.15.142:0/2723448437 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7090107d50 0x7f7090199f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:09.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:09.200+0000 7f70977f6640 1 --2- 172.21.15.142:0/2723448437 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f70901073d0 0x7f70901999d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:09.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:09.200+0000 7f70977f6640 1 --2- 172.21.15.142:0/2723448437 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f70901019c0 0x7f7090199490 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:09.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:09.200+0000 7f70977f6640 1 -- 172.21.15.142:0/2723448437 >> 172.21.15.142:0/2723448437 conn(0x7f70900fd430 msgr2=0x7f70900ff820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:28:09.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:09.200+0000 7f70977f6640 1 -- 172.21.15.142:0/2723448437 shutdown_connections 2024-09-06T14:28:09.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:09.200+0000 7f70977f6640 1 -- 172.21.15.142:0/2723448437 wait complete. 2024-09-06T14:28:09.298 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:28:09 smithi162 ceph-mon[33787]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-06T14:28:09.298 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:28:09 smithi162 ceph-mon[33787]: pgmap v101: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:28:10.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:28:10 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/2723448437' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:28:10.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:28:10 smithi142 ceph-mon[37372]: from='client.? 172.21.15.142:0/2723448437' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:28:10.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:28:10 smithi162 ceph-mon[33787]: from='client.? 172.21.15.142:0/2723448437' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:28:11.298 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:28:11 smithi162 ceph-mon[33787]: pgmap v102: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:28:11.298 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:28:11 smithi162 ceph-mon[33787]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-06T14:28:11.459 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:28:11 smithi142 ceph-mon[29958]: pgmap v102: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:28:11.459 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:28:11 smithi142 ceph-mon[29958]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-06T14:28:11.459 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:28:11 smithi142 ceph-mon[37372]: pgmap v102: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:28:11.459 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:28:11 smithi142 ceph-mon[37372]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-06T14:28:12.844 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph health --format=json 2024-09-06T14:28:13.115 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:28:13.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:28:13 smithi142 ceph-mon[37372]: pgmap v103: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:28:13.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:28:13 smithi142 ceph-mon[37372]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-06T14:28:13.210 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:28:13 smithi142 ceph-mon[29958]: pgmap v103: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:28:13.210 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:28:13 smithi142 ceph-mon[29958]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-06T14:28:13.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:28:13 smithi162 ceph-mon[33787]: pgmap v103: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:28:13.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:28:13 smithi162 ceph-mon[33787]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-06T14:28:13.618 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:13.594+0000 7f07be929640 1 -- 172.21.15.142:0/4101740846 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f07b81039d0 msgr2=0x7f07b8103db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:28:13.618 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:13.594+0000 7f07be929640 1 --2- 172.21.15.142:0/4101740846 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f07b81039d0 0x7f07b8103db0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f07a8009900 tx=0x7f07a802a410 comp rx=0 tx=0).stop 2024-09-06T14:28:13.619 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:13.594+0000 7f07be929640 1 -- 172.21.15.142:0/4101740846 shutdown_connections 2024-09-06T14:28:13.619 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:13.594+0000 7f07be929640 1 --2- 172.21.15.142:0/4101740846 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f07b810ce30 0x7f07b810f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:13.619 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:13.594+0000 7f07be929640 1 --2- 172.21.15.142:0/4101740846 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f07b81042f0 0x7f07b810c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:13.619 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:13.594+0000 7f07be929640 1 --2- 172.21.15.142:0/4101740846 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f07b81039d0 0x7f07b8103db0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:13.619 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:13.594+0000 7f07be929640 1 -- 172.21.15.142:0/4101740846 >> 172.21.15.142:0/4101740846 conn(0x7f07b80fd430 msgr2=0x7f07b80ff850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:28:13.619 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:13.594+0000 7f07be929640 1 -- 172.21.15.142:0/4101740846 shutdown_connections 2024-09-06T14:28:13.619 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:13.594+0000 7f07be929640 1 -- 172.21.15.142:0/4101740846 wait complete. 2024-09-06T14:28:13.619 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:13.594+0000 7f07be929640 1 Processor -- start 2024-09-06T14:28:13.619 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:13.594+0000 7f07be929640 1 -- start start 2024-09-06T14:28:13.619 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:13.595+0000 7f07be929640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f07b81039d0 0x7f07b81986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:28:13.619 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:13.595+0000 7f07be929640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f07b81042f0 0x7f07b8198c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:28:13.619 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:13.595+0000 7f07be929640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f07b810ce30 0x7f07b819dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:28:13.619 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:13.595+0000 7f07be929640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f07b81121b0 con 0x7f07b81039d0 2024-09-06T14:28:13.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:13.595+0000 7f07be929640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f07b8112030 con 0x7f07b810ce30 2024-09-06T14:28:13.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:13.595+0000 7f07be929640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7f07b8112330 con 0x7f07b81042f0 2024-09-06T14:28:13.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:13.595+0000 7f07b77fe640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f07b81042f0 0x7f07b8198c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:28:13.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:13.595+0000 7f07b7fff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f07b81039d0 0x7f07b81986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:28:13.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:13.595+0000 7f07b77fe640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f07b81042f0 0x7f07b8198c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:34340/0 (socket says 172.21.15.142:34340) 2024-09-06T14:28:13.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:13.595+0000 7f07b7fff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f07b81039d0 0x7f07b81986e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:46368/0 (socket says 172.21.15.142:46368) 2024-09-06T14:28:13.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:13.595+0000 7f07b77fe640 1 -- 172.21.15.142:0/505823023 learned_addr learned my addr 172.21.15.142:0/505823023 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:28:13.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:13.595+0000 7f07bcec6640 1 --2- 172.21.15.142:0/505823023 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f07b810ce30 0x7f07b819dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:28:13.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:13.595+0000 7f07b77fe640 1 -- 172.21.15.142:0/505823023 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f07b810ce30 msgr2=0x7f07b819dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:28:13.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:13.595+0000 7f07b77fe640 1 --2- 172.21.15.142:0/505823023 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f07b810ce30 0x7f07b819dfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:13.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:13.595+0000 7f07b77fe640 1 -- 172.21.15.142:0/505823023 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f07b81039d0 msgr2=0x7f07b81986e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:28:13.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:13.595+0000 7f07b77fe640 1 --2- 172.21.15.142:0/505823023 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f07b81039d0 0x7f07b81986e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:13.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:13.595+0000 7f07b77fe640 1 -- 172.21.15.142:0/505823023 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07b819e790 con 0x7f07b81042f0 2024-09-06T14:28:13.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:13.595+0000 7f07b7fff640 1 --2- 172.21.15.142:0/505823023 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f07b81039d0 0x7f07b81986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-06T14:28:13.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:13.595+0000 7f07b77fe640 1 --2- 172.21.15.142:0/505823023 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f07b81042f0 0x7f07b8198c20 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f07ac00e9c0 tx=0x7f07ac00edc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:28:13.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:13.595+0000 7f07b57fa640 1 -- 172.21.15.142:0/505823023 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07ac00cd80 con 0x7f07b81042f0 2024-09-06T14:28:13.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:13.595+0000 7f07be929640 1 -- 172.21.15.142:0/505823023 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07b819ea80 con 0x7f07b81042f0 2024-09-06T14:28:13.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:13.595+0000 7f07b57fa640 1 -- 172.21.15.142:0/505823023 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7f07ac0096c0 con 0x7f07b81042f0 2024-09-06T14:28:13.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:13.595+0000 7f07b57fa640 1 -- 172.21.15.142:0/505823023 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07ac018700 con 0x7f07b81042f0 2024-09-06T14:28:13.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:13.596+0000 7f07be929640 1 -- 172.21.15.142:0/505823023 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f07b81a5310 con 0x7f07b81042f0 2024-09-06T14:28:13.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:13.596+0000 7f07be929640 1 -- 172.21.15.142:0/505823023 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07b806aee0 con 0x7f07b81042f0 2024-09-06T14:28:13.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:13.597+0000 7f07b57fa640 1 -- 172.21.15.142:0/505823023 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 22) ==== 99844+0+0 (secure 0 0 0) 0x7f07ac01f020 con 0x7f07b81042f0 2024-09-06T14:28:13.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:13.598+0000 7f07b57fa640 1 --2- 172.21.15.142:0/505823023 >> v2:172.21.15.142:6800/2944545612 conn(0x7f07a0077620 0x7f07a0079ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:28:13.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:13.598+0000 7f07b57fa640 1 -- 172.21.15.142:0/505823023 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(64..64 src has 1..64) ==== 6130+0+0 (secure 0 0 0) 0x7f07ac099850 con 0x7f07b81042f0 2024-09-06T14:28:13.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:13.598+0000 7f07b7fff640 1 --2- 172.21.15.142:0/505823023 >> v2:172.21.15.142:6800/2944545612 conn(0x7f07a0077620 0x7f07a0079ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:28:13.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:13.598+0000 7f07b7fff640 1 --2- 172.21.15.142:0/505823023 >> v2:172.21.15.142:6800/2944545612 conn(0x7f07a0077620 0x7f07a0079ae0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f07a8031440 tx=0x7f07a8031850 comp rx=0 tx=0).ready entity=mgr.14601 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:28:13.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:13.602+0000 7f07b57fa640 1 -- 172.21.15.142:0/505823023 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7f07ac066a80 con 0x7f07b81042f0 2024-09-06T14:28:13.793 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:13.796+0000 7f07be929640 1 -- 172.21.15.142:0/505823023 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f07b8199810 con 0x7f07b81042f0 2024-09-06T14:28:13.793 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:13.796+0000 7f07b57fa640 1 -- 172.21.15.142:0/505823023 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+212 (secure 0 0 0) 0x7f07ac06b4a0 con 0x7f07b81042f0 2024-09-06T14:28:13.794 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:28:13.794 INFO:teuthology.orchestra.run.smithi142.stdout:{"status":"HEALTH_WARN","checks":{"MGR_MODULE_DEPENDENCY":{"severity":"HEALTH_WARN","summary":{"message":"Module 'volumes' has failed dependency: No module named 'ceph.fs'","count":1},"muted":false}},"mutes":[]} 2024-09-06T14:28:13.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:13.798+0000 7f07be929640 1 -- 172.21.15.142:0/505823023 >> v2:172.21.15.142:6800/2944545612 conn(0x7f07a0077620 msgr2=0x7f07a0079ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:28:13.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:13.798+0000 7f07be929640 1 --2- 172.21.15.142:0/505823023 >> v2:172.21.15.142:6800/2944545612 conn(0x7f07a0077620 0x7f07a0079ae0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f07a8031440 tx=0x7f07a8031850 comp rx=0 tx=0).stop 2024-09-06T14:28:13.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:13.798+0000 7f07be929640 1 -- 172.21.15.142:0/505823023 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f07b81042f0 msgr2=0x7f07b8198c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:28:13.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:13.798+0000 7f07be929640 1 --2- 172.21.15.142:0/505823023 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f07b81042f0 0x7f07b8198c20 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f07ac00e9c0 tx=0x7f07ac00edc0 comp rx=0 tx=0).stop 2024-09-06T14:28:13.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:13.798+0000 7f07be929640 1 -- 172.21.15.142:0/505823023 shutdown_connections 2024-09-06T14:28:13.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:13.798+0000 7f07be929640 1 --2- 172.21.15.142:0/505823023 >> v2:172.21.15.142:6800/2944545612 conn(0x7f07a0077620 0x7f07a0079ae0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:13.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:13.798+0000 7f07be929640 1 --2- 172.21.15.142:0/505823023 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f07b810ce30 0x7f07b819dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:13.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:13.798+0000 7f07be929640 1 --2- 172.21.15.142:0/505823023 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f07b81042f0 0x7f07b8198c20 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:13.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:13.798+0000 7f07be929640 1 --2- 172.21.15.142:0/505823023 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f07b81039d0 0x7f07b81986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:13.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:13.798+0000 7f07be929640 1 -- 172.21.15.142:0/505823023 >> 172.21.15.142:0/505823023 conn(0x7f07b80fd430 msgr2=0x7f07b80fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:28:13.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:13.798+0000 7f07be929640 1 -- 172.21.15.142:0/505823023 shutdown_connections 2024-09-06T14:28:13.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:13.798+0000 7f07be929640 1 -- 172.21.15.142:0/505823023 wait complete. 2024-09-06T14:28:14.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:28:14 smithi142 ceph-mon[29958]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:28:14.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:28:14 smithi142 ceph-mon[29958]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-06T14:28:14.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:28:14 smithi142 ceph-mon[29958]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:28:14.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:28:14 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/505823023' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:28:14.210 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:28:14 smithi142 ceph-mon[37372]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:28:14.210 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:28:14 smithi142 ceph-mon[37372]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-06T14:28:14.210 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:28:14 smithi142 ceph-mon[37372]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:28:14.210 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:28:14 smithi142 ceph-mon[37372]: from='client.? 172.21.15.142:0/505823023' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:28:14.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:28:14 smithi162 ceph-mon[33787]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:28:14.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:28:14 smithi162 ceph-mon[33787]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-06T14:28:14.298 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:28:14 smithi162 ceph-mon[33787]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:28:14.298 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:28:14 smithi162 ceph-mon[33787]: from='client.? 172.21.15.142:0/505823023' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:28:14.709 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:28:14 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: ::ffff:172.21.15.162 - - [06/Sep/2024:14:28:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-06T14:28:15.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:28:15 smithi162 ceph-mon[33787]: pgmap v104: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:28:15.459 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:28:15 smithi142 ceph-mon[37372]: pgmap v104: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:28:15.459 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:28:15 smithi142 ceph-mon[29958]: pgmap v104: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:28:17.297 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 06 14:28:17 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-iscsi-iscsi-a[59853]: debug there is no tcmu-runner data available 2024-09-06T14:28:17.298 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:28:17 smithi162 ceph-mon[33787]: pgmap v105: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:28:17.443 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph health --format=json 2024-09-06T14:28:17.459 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:28:17 smithi142 ceph-mon[29958]: pgmap v105: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:28:17.459 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:28:17 smithi142 ceph-mon[37372]: pgmap v105: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:28:17.705 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:28:18.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:18.164+0000 7fb40d08e640 1 -- 172.21.15.142:0/273518094 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb408107d90 msgr2=0x7fb40810e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:28:18.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:18.164+0000 7fb40d08e640 1 --2- 172.21.15.142:0/273518094 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb408107d90 0x7fb40810e590 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fb400009ca0 tx=0x7fb40002a8c0 comp rx=0 tx=0).stop 2024-09-06T14:28:18.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:18.164+0000 7fb40d08e640 1 -- 172.21.15.142:0/273518094 shutdown_connections 2024-09-06T14:28:18.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:18.164+0000 7fb40d08e640 1 --2- 172.21.15.142:0/273518094 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb408107d90 0x7fb40810e590 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:18.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:18.164+0000 7fb40d08e640 1 --2- 172.21.15.142:0/273518094 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb408107410 0x7fb408107850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:18.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:18.164+0000 7fb40d08e640 1 --2- 172.21.15.142:0/273518094 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb408101a00 0x7fb408101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:18.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:18.164+0000 7fb40d08e640 1 -- 172.21.15.142:0/273518094 >> 172.21.15.142:0/273518094 conn(0x7fb4080fd470 msgr2=0x7fb4080ff890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:28:18.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:18.164+0000 7fb40d08e640 1 -- 172.21.15.142:0/273518094 shutdown_connections 2024-09-06T14:28:18.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:18.164+0000 7fb40d08e640 1 -- 172.21.15.142:0/273518094 wait complete. 2024-09-06T14:28:18.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:18.165+0000 7fb40d08e640 1 Processor -- start 2024-09-06T14:28:18.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:18.165+0000 7fb40d08e640 1 -- start start 2024-09-06T14:28:18.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:18.165+0000 7fb40d08e640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb408101a00 0x7fb408199470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:28:18.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:18.165+0000 7fb40d08e640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb408107410 0x7fb4081999b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:28:18.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:18.165+0000 7fb40d08e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb408107d90 0x7fb408199f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:28:18.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:18.165+0000 7fb40d08e640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fb408111160 con 0x7fb408101a00 2024-09-06T14:28:18.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:18.165+0000 7fb40d08e640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb408110fe0 con 0x7fb408107d90 2024-09-06T14:28:18.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:18.165+0000 7fb40d08e640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7fb4081112e0 con 0x7fb408107410 2024-09-06T14:28:18.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:18.165+0000 7fb3fe59c640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb408107410 0x7fb4081999b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:28:18.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:18.165+0000 7fb406d9d640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb408101a00 0x7fb408199470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:28:18.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:18.165+0000 7fb3fe59c640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb408107410 0x7fb4081999b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:56416/0 (socket says 172.21.15.142:56416) 2024-09-06T14:28:18.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:18.165+0000 7fb3fe59c640 1 -- 172.21.15.142:0/2215839652 learned_addr learned my addr 172.21.15.142:0/2215839652 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:28:18.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:18.165+0000 7fb40759e640 1 --2- 172.21.15.142:0/2215839652 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb408107d90 0x7fb408199f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:28:18.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:18.166+0000 7fb406d9d640 1 -- 172.21.15.142:0/2215839652 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb408107410 msgr2=0x7fb4081999b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:28:18.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:18.166+0000 7fb406d9d640 1 --2- 172.21.15.142:0/2215839652 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb408107410 0x7fb4081999b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:18.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:18.166+0000 7fb406d9d640 1 -- 172.21.15.142:0/2215839652 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb408107d90 msgr2=0x7fb408199f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:28:18.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:18.166+0000 7fb406d9d640 1 --2- 172.21.15.142:0/2215839652 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb408107d90 0x7fb408199f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:18.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:18.166+0000 7fb406d9d640 1 -- 172.21.15.142:0/2215839652 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4081a4870 con 0x7fb408101a00 2024-09-06T14:28:18.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:18.166+0000 7fb40759e640 1 --2- 172.21.15.142:0/2215839652 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb408107d90 0x7fb408199f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-06T14:28:18.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:18.166+0000 7fb406d9d640 1 --2- 172.21.15.142:0/2215839652 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb408101a00 0x7fb408199470 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fb3f800d6d0 tx=0x7fb3f800db90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:28:18.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:18.166+0000 7fb404d99640 1 -- 172.21.15.142:0/2215839652 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3f8017070 con 0x7fb408101a00 2024-09-06T14:28:18.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:18.166+0000 7fb404d99640 1 -- 172.21.15.142:0/2215839652 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7fb3f8009820 con 0x7fb408101a00 2024-09-06T14:28:18.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:18.166+0000 7fb40d08e640 1 -- 172.21.15.142:0/2215839652 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4081a4b60 con 0x7fb408101a00 2024-09-06T14:28:18.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:18.166+0000 7fb404d99640 1 -- 172.21.15.142:0/2215839652 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3f8004b80 con 0x7fb408101a00 2024-09-06T14:28:18.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:18.166+0000 7fb40d08e640 1 -- 172.21.15.142:0/2215839652 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4081a50a0 con 0x7fb408101a00 2024-09-06T14:28:18.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:18.167+0000 7fb40d08e640 1 -- 172.21.15.142:0/2215839652 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb408101de0 con 0x7fb408101a00 2024-09-06T14:28:18.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:18.167+0000 7fb404d99640 1 -- 172.21.15.142:0/2215839652 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 22) ==== 99844+0+0 (secure 0 0 0) 0x7fb3f8005220 con 0x7fb408101a00 2024-09-06T14:28:18.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:18.168+0000 7fb404d99640 1 --2- 172.21.15.142:0/2215839652 >> v2:172.21.15.142:6800/2944545612 conn(0x7fb3f4077620 0x7fb3f4079ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:28:18.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:18.168+0000 7fb404d99640 1 -- 172.21.15.142:0/2215839652 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 6130+0+0 (secure 0 0 0) 0x7fb3f80997b0 con 0x7fb408101a00 2024-09-06T14:28:18.187 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:18.168+0000 7fb3fe59c640 1 --2- 172.21.15.142:0/2215839652 >> v2:172.21.15.142:6800/2944545612 conn(0x7fb3f4077620 0x7fb3f4079ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:28:18.187 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:18.168+0000 7fb3fe59c640 1 --2- 172.21.15.142:0/2215839652 >> v2:172.21.15.142:6800/2944545612 conn(0x7fb3f4077620 0x7fb3f4079ae0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fb3f00095f0 tx=0x7fb3f0005f90 comp rx=0 tx=0).ready entity=mgr.14601 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:28:18.187 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:18.172+0000 7fb404d99640 1 -- 172.21.15.142:0/2215839652 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7fb3f80669e0 con 0x7fb408101a00 2024-09-06T14:28:18.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:18.359+0000 7fb40d08e640 1 -- 172.21.15.142:0/2215839652 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fb40802e370 con 0x7fb408101a00 2024-09-06T14:28:18.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:18.360+0000 7fb404d99640 1 -- 172.21.15.142:0/2215839652 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+212 (secure 0 0 0) 0x7fb3f806b400 con 0x7fb408101a00 2024-09-06T14:28:18.359 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:28:18.359 INFO:teuthology.orchestra.run.smithi142.stdout:{"status":"HEALTH_WARN","checks":{"MGR_MODULE_DEPENDENCY":{"severity":"HEALTH_WARN","summary":{"message":"Module 'volumes' has failed dependency: No module named 'ceph.fs'","count":1},"muted":false}},"mutes":[]} 2024-09-06T14:28:18.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:18.362+0000 7fb40d08e640 1 -- 172.21.15.142:0/2215839652 >> v2:172.21.15.142:6800/2944545612 conn(0x7fb3f4077620 msgr2=0x7fb3f4079ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:28:18.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:18.362+0000 7fb40d08e640 1 --2- 172.21.15.142:0/2215839652 >> v2:172.21.15.142:6800/2944545612 conn(0x7fb3f4077620 0x7fb3f4079ae0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fb3f00095f0 tx=0x7fb3f0005f90 comp rx=0 tx=0).stop 2024-09-06T14:28:18.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:18.362+0000 7fb40d08e640 1 -- 172.21.15.142:0/2215839652 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb408101a00 msgr2=0x7fb408199470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:28:18.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:18.362+0000 7fb40d08e640 1 --2- 172.21.15.142:0/2215839652 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb408101a00 0x7fb408199470 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fb3f800d6d0 tx=0x7fb3f800db90 comp rx=0 tx=0).stop 2024-09-06T14:28:18.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:18.362+0000 7fb40d08e640 1 -- 172.21.15.142:0/2215839652 shutdown_connections 2024-09-06T14:28:18.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:18.362+0000 7fb40d08e640 1 --2- 172.21.15.142:0/2215839652 >> v2:172.21.15.142:6800/2944545612 conn(0x7fb3f4077620 0x7fb3f4079ae0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:18.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:18.362+0000 7fb40d08e640 1 --2- 172.21.15.142:0/2215839652 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb408107d90 0x7fb408199f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:18.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:18.362+0000 7fb40d08e640 1 --2- 172.21.15.142:0/2215839652 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb408107410 0x7fb4081999b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:18.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:18.362+0000 7fb40d08e640 1 --2- 172.21.15.142:0/2215839652 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb408101a00 0x7fb408199470 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:18.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:18.362+0000 7fb40d08e640 1 -- 172.21.15.142:0/2215839652 >> 172.21.15.142:0/2215839652 conn(0x7fb4080fd470 msgr2=0x7fb4080ff860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:28:18.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:18.362+0000 7fb40d08e640 1 -- 172.21.15.142:0/2215839652 shutdown_connections 2024-09-06T14:28:18.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:18.362+0000 7fb40d08e640 1 -- 172.21.15.142:0/2215839652 wait complete. 2024-09-06T14:28:19.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:28:19 smithi142 ceph-mon[37372]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-06T14:28:19.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:28:19 smithi142 ceph-mon[37372]: pgmap v106: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:28:19.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:28:19 smithi142 ceph-mon[37372]: from='client.? 172.21.15.142:0/2215839652' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:28:19.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:28:19 smithi142 ceph-mon[29958]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-06T14:28:19.210 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:28:19 smithi142 ceph-mon[29958]: pgmap v106: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:28:19.210 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:28:19 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/2215839652' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:28:19.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:28:19 smithi162 ceph-mon[33787]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-06T14:28:19.298 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:28:19 smithi162 ceph-mon[33787]: pgmap v106: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:28:19.298 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:28:19 smithi162 ceph-mon[33787]: from='client.? 172.21.15.142:0/2215839652' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:28:21.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:28:21 smithi162 ceph-mon[33787]: pgmap v107: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:28:21.459 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:28:21 smithi142 ceph-mon[29958]: pgmap v107: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:28:21.459 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:28:21 smithi142 ceph-mon[37372]: pgmap v107: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:28:22.074 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph health --format=json 2024-09-06T14:28:22.334 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:28:22.832 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:22.821+0000 7f63935e5640 1 -- 172.21.15.142:0/2057430692 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f638c108e70 msgr2=0x7f638c10f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:28:22.832 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:22.821+0000 7f63935e5640 1 --2- 172.21.15.142:0/2057430692 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f638c108e70 0x7f638c10f670 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f6384009ca0 tx=0x7f638402a8c0 comp rx=0 tx=0).stop 2024-09-06T14:28:22.832 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:22.821+0000 7f63935e5640 1 -- 172.21.15.142:0/2057430692 shutdown_connections 2024-09-06T14:28:22.832 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:22.821+0000 7f63935e5640 1 --2- 172.21.15.142:0/2057430692 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f638c108e70 0x7f638c10f670 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:22.832 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:22.821+0000 7f63935e5640 1 --2- 172.21.15.142:0/2057430692 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f638c1084f0 0x7f638c108930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:22.832 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:22.821+0000 7f63935e5640 1 --2- 172.21.15.142:0/2057430692 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f638c102ae0 0x7f638c102ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:22.832 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:22.821+0000 7f63935e5640 1 -- 172.21.15.142:0/2057430692 >> 172.21.15.142:0/2057430692 conn(0x7f638c0fe530 msgr2=0x7f638c100950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:28:22.832 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:22.821+0000 7f63935e5640 1 -- 172.21.15.142:0/2057430692 shutdown_connections 2024-09-06T14:28:22.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:22.821+0000 7f63935e5640 1 -- 172.21.15.142:0/2057430692 wait complete. 2024-09-06T14:28:22.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:22.822+0000 7f63935e5640 1 Processor -- start 2024-09-06T14:28:22.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:22.822+0000 7f63935e5640 1 -- start start 2024-09-06T14:28:22.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:22.822+0000 7f63935e5640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f638c102ae0 0x7f638c198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:28:22.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:22.822+0000 7f63935e5640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f638c1084f0 0x7f638c198b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:28:22.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:22.822+0000 7f63935e5640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f638c108e70 0x7f638c19df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:28:22.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:22.822+0000 7f63935e5640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f638c112200 con 0x7f638c108e70 2024-09-06T14:28:22.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:22.822+0000 7f63935e5640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f638c112080 con 0x7f638c102ae0 2024-09-06T14:28:22.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:22.822+0000 7f63935e5640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7f638c112380 con 0x7f638c1084f0 2024-09-06T14:28:22.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:22.822+0000 7f6390b80640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f638c1084f0 0x7f638c198b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:28:22.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:22.822+0000 7f6391b82640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f638c108e70 0x7f638c19df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:28:22.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:22.822+0000 7f6390b80640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f638c1084f0 0x7f638c198b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:56436/0 (socket says 172.21.15.142:56436) 2024-09-06T14:28:22.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:22.822+0000 7f6390b80640 1 -- 172.21.15.142:0/1263859288 learned_addr learned my addr 172.21.15.142:0/1263859288 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:28:22.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:22.822+0000 7f6391381640 1 --2- 172.21.15.142:0/1263859288 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f638c102ae0 0x7f638c198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:28:22.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:22.823+0000 7f6390b80640 1 -- 172.21.15.142:0/1263859288 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f638c102ae0 msgr2=0x7f638c198630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:28:22.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:22.823+0000 7f6390b80640 1 --2- 172.21.15.142:0/1263859288 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f638c102ae0 0x7f638c198630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:22.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:22.823+0000 7f6390b80640 1 -- 172.21.15.142:0/1263859288 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f638c108e70 msgr2=0x7f638c19df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:28:22.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:22.823+0000 7f6390b80640 1 --2- 172.21.15.142:0/1263859288 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f638c108e70 0x7f638c19df10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:22.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:22.823+0000 7f6390b80640 1 -- 172.21.15.142:0/1263859288 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f638c19e6e0 con 0x7f638c1084f0 2024-09-06T14:28:22.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:22.823+0000 7f6391b82640 1 --2- 172.21.15.142:0/1263859288 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f638c108e70 0x7f638c19df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-06T14:28:22.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:22.823+0000 7f6390b80640 1 --2- 172.21.15.142:0/1263859288 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f638c1084f0 0x7f638c198b70 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f637c00d950 tx=0x7f637c00de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:28:22.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:22.823+0000 7f63827fc640 1 -- 172.21.15.142:0/1263859288 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f637c017070 con 0x7f638c1084f0 2024-09-06T14:28:22.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:22.823+0000 7f63827fc640 1 -- 172.21.15.142:0/1263859288 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7f637c009c30 con 0x7f638c1084f0 2024-09-06T14:28:22.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:22.823+0000 7f63935e5640 1 -- 172.21.15.142:0/1263859288 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f638c19e9d0 con 0x7f638c1084f0 2024-09-06T14:28:22.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:22.823+0000 7f63827fc640 1 -- 172.21.15.142:0/1263859288 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f637c00f620 con 0x7f638c1084f0 2024-09-06T14:28:22.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:22.823+0000 7f63935e5640 1 -- 172.21.15.142:0/1263859288 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f638c1a5090 con 0x7f638c1084f0 2024-09-06T14:28:22.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:22.824+0000 7f63935e5640 1 -- 172.21.15.142:0/1263859288 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f638c102ec0 con 0x7f638c1084f0 2024-09-06T14:28:22.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:22.825+0000 7f63827fc640 1 -- 172.21.15.142:0/1263859288 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 22) ==== 99844+0+0 (secure 0 0 0) 0x7f637c010460 con 0x7f638c1084f0 2024-09-06T14:28:22.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:22.825+0000 7f63827fc640 1 --2- 172.21.15.142:0/1263859288 >> v2:172.21.15.142:6800/2944545612 conn(0x7f6378077620 0x7f6378079ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:28:22.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:22.825+0000 7f63827fc640 1 -- 172.21.15.142:0/1263859288 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(64..64 src has 1..64) ==== 6130+0+0 (secure 0 0 0) 0x7f637c099590 con 0x7f638c1084f0 2024-09-06T14:28:22.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:22.825+0000 7f6391381640 1 --2- 172.21.15.142:0/1263859288 >> v2:172.21.15.142:6800/2944545612 conn(0x7f6378077620 0x7f6378079ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:28:22.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:22.826+0000 7f6391381640 1 --2- 172.21.15.142:0/1263859288 >> v2:172.21.15.142:6800/2944545612 conn(0x7f6378077620 0x7f6378079ae0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f63880095f0 tx=0x7f6388005f90 comp rx=0 tx=0).ready entity=mgr.14601 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:28:22.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:22.829+0000 7f63827fc640 1 -- 172.21.15.142:0/1263859288 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7f637c0667c0 con 0x7f638c1084f0 2024-09-06T14:28:23.015 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:23.018+0000 7f63935e5640 1 -- 172.21.15.142:0/1263859288 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f638c10ac50 con 0x7f638c1084f0 2024-09-06T14:28:23.015 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:28:23.016 INFO:teuthology.orchestra.run.smithi142.stdout:{"status":"HEALTH_WARN","checks":{"MGR_MODULE_DEPENDENCY":{"severity":"HEALTH_WARN","summary":{"message":"Module 'volumes' has failed dependency: No module named 'ceph.fs'","count":1},"muted":false}},"mutes":[]} 2024-09-06T14:28:23.016 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:23.018+0000 7f63827fc640 1 -- 172.21.15.142:0/1263859288 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+212 (secure 0 0 0) 0x7f637c06b1e0 con 0x7f638c1084f0 2024-09-06T14:28:23.018 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:23.020+0000 7f63935e5640 1 -- 172.21.15.142:0/1263859288 >> v2:172.21.15.142:6800/2944545612 conn(0x7f6378077620 msgr2=0x7f6378079ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:28:23.018 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:23.020+0000 7f63935e5640 1 --2- 172.21.15.142:0/1263859288 >> v2:172.21.15.142:6800/2944545612 conn(0x7f6378077620 0x7f6378079ae0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f63880095f0 tx=0x7f6388005f90 comp rx=0 tx=0).stop 2024-09-06T14:28:23.018 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:23.020+0000 7f63935e5640 1 -- 172.21.15.142:0/1263859288 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f638c1084f0 msgr2=0x7f638c198b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:28:23.018 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:23.020+0000 7f63935e5640 1 --2- 172.21.15.142:0/1263859288 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f638c1084f0 0x7f638c198b70 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f637c00d950 tx=0x7f637c00de10 comp rx=0 tx=0).stop 2024-09-06T14:28:23.018 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:23.020+0000 7f63935e5640 1 -- 172.21.15.142:0/1263859288 shutdown_connections 2024-09-06T14:28:23.018 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:23.020+0000 7f63935e5640 1 --2- 172.21.15.142:0/1263859288 >> v2:172.21.15.142:6800/2944545612 conn(0x7f6378077620 0x7f6378079ae0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:23.018 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:23.020+0000 7f63935e5640 1 --2- 172.21.15.142:0/1263859288 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f638c108e70 0x7f638c19df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:23.018 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:23.020+0000 7f63935e5640 1 --2- 172.21.15.142:0/1263859288 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f638c1084f0 0x7f638c198b70 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:23.018 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:23.020+0000 7f63935e5640 1 --2- 172.21.15.142:0/1263859288 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f638c102ae0 0x7f638c198630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:23.018 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:23.020+0000 7f63935e5640 1 -- 172.21.15.142:0/1263859288 >> 172.21.15.142:0/1263859288 conn(0x7f638c0fe530 msgr2=0x7f638c0ffdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:28:23.018 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:23.020+0000 7f63935e5640 1 -- 172.21.15.142:0/1263859288 shutdown_connections 2024-09-06T14:28:23.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:23.020+0000 7f63935e5640 1 -- 172.21.15.142:0/1263859288 wait complete. 2024-09-06T14:28:23.293 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:28:23 smithi142 ceph-mon[29958]: pgmap v108: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:28:23.294 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:28:23 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/1263859288' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:28:23.294 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:28:23 smithi142 ceph-mon[37372]: pgmap v108: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:28:23.294 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:28:23 smithi142 ceph-mon[37372]: from='client.? 172.21.15.142:0/1263859288' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:28:23.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:28:23 smithi162 ceph-mon[33787]: pgmap v108: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:28:23.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:28:23 smithi162 ceph-mon[33787]: from='client.? 172.21.15.142:0/1263859288' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:28:24.709 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:28:24 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: ::ffff:172.21.15.162 - - [06/Sep/2024:14:28:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-06T14:28:25.298 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:28:25 smithi162 ceph-mon[33787]: pgmap v109: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:28:25.459 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:28:25 smithi142 ceph-mon[37372]: pgmap v109: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:28:25.459 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:28:25 smithi142 ceph-mon[29958]: pgmap v109: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:28:26.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:28:26 smithi162 ceph-mon[33787]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-06T14:28:26.459 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:28:26 smithi142 ceph-mon[29958]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-06T14:28:26.459 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:28:26 smithi142 ceph-mon[37372]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-06T14:28:26.707 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph health --format=json 2024-09-06T14:28:26.970 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:28:27.297 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 06 14:28:27 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-iscsi-iscsi-a[59853]: debug there is no tcmu-runner data available 2024-09-06T14:28:27.298 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:28:27 smithi162 ceph-mon[33787]: pgmap v110: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:28:27.300 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:28:27 smithi142 ceph-mon[29958]: pgmap v110: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:28:27.300 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:28:27 smithi142 ceph-mon[37372]: pgmap v110: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:28:27.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:27.508+0000 7f9cbce6c640 1 -- 172.21.15.142:0/2903710860 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9cb81084f0 msgr2=0x7f9cb8108930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:28:27.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:27.508+0000 7f9cbce6c640 1 --2- 172.21.15.142:0/2903710860 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9cb81084f0 0x7f9cb8108930 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f9cac009860 tx=0x7f9cac02a340 comp rx=0 tx=0).stop 2024-09-06T14:28:27.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:27.508+0000 7f9cbce6c640 1 -- 172.21.15.142:0/2903710860 shutdown_connections 2024-09-06T14:28:27.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:27.508+0000 7f9cbce6c640 1 --2- 172.21.15.142:0/2903710860 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9cb8108e70 0x7f9cb810f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:27.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:27.508+0000 7f9cbce6c640 1 --2- 172.21.15.142:0/2903710860 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9cb81084f0 0x7f9cb8108930 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:27.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:27.508+0000 7f9cbce6c640 1 --2- 172.21.15.142:0/2903710860 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9cb8102ae0 0x7f9cb8102ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:27.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:27.508+0000 7f9cbce6c640 1 -- 172.21.15.142:0/2903710860 >> 172.21.15.142:0/2903710860 conn(0x7f9cb80fe550 msgr2=0x7f9cb8100970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:28:27.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:27.508+0000 7f9cbce6c640 1 -- 172.21.15.142:0/2903710860 shutdown_connections 2024-09-06T14:28:27.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:27.508+0000 7f9cbce6c640 1 -- 172.21.15.142:0/2903710860 wait complete. 2024-09-06T14:28:27.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:27.509+0000 7f9cbce6c640 1 Processor -- start 2024-09-06T14:28:27.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:27.509+0000 7f9cbce6c640 1 -- start start 2024-09-06T14:28:27.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:27.509+0000 7f9cbce6c640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9cb8102ae0 0x7f9cb8198690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:28:27.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:27.509+0000 7f9cbce6c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9cb81084f0 0x7f9cb8198bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:28:27.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:27.509+0000 7f9cbce6c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9cb8108e70 0x7f9cb819df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:28:27.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:27.509+0000 7f9cbce6c640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f9cb81121c0 con 0x7f9cb81084f0 2024-09-06T14:28:27.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:27.509+0000 7f9cbce6c640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f9cb8112040 con 0x7f9cb8108e70 2024-09-06T14:28:27.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:27.509+0000 7f9cbce6c640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7f9cb8112340 con 0x7f9cb8102ae0 2024-09-06T14:28:27.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:27.509+0000 7f9cb5d9b640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9cb81084f0 0x7f9cb8198bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:28:27.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:27.509+0000 7f9cb659c640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9cb8102ae0 0x7f9cb8198690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:28:27.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:27.509+0000 7f9cb5d9b640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9cb81084f0 0x7f9cb8198bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:39266/0 (socket says 172.21.15.142:39266) 2024-09-06T14:28:27.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:27.509+0000 7f9cb5d9b640 1 -- 172.21.15.142:0/1854945191 learned_addr learned my addr 172.21.15.142:0/1854945191 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:28:27.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:27.510+0000 7f9cb6d9d640 1 --2- 172.21.15.142:0/1854945191 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9cb8108e70 0x7f9cb819df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:28:27.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:27.510+0000 7f9cb659c640 1 -- 172.21.15.142:0/1854945191 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9cb8108e70 msgr2=0x7f9cb819df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:28:27.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:27.510+0000 7f9cb659c640 1 --2- 172.21.15.142:0/1854945191 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9cb8108e70 0x7f9cb819df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:27.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:27.510+0000 7f9cb659c640 1 -- 172.21.15.142:0/1854945191 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9cb81084f0 msgr2=0x7f9cb8198bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:28:27.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:27.510+0000 7f9cb659c640 1 --2- 172.21.15.142:0/1854945191 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9cb81084f0 0x7f9cb8198bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:27.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:27.510+0000 7f9cb659c640 1 -- 172.21.15.142:0/1854945191 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cb819e740 con 0x7f9cb8102ae0 2024-09-06T14:28:27.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:27.510+0000 7f9cb6d9d640 1 --2- 172.21.15.142:0/1854945191 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9cb8108e70 0x7f9cb819df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-06T14:28:27.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:27.510+0000 7f9cb5d9b640 1 --2- 172.21.15.142:0/1854945191 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9cb81084f0 0x7f9cb8198bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-06T14:28:27.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:27.510+0000 7f9cb659c640 1 --2- 172.21.15.142:0/1854945191 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9cb8102ae0 0x7f9cb8198690 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f9ca8005fd0 tx=0x7f9ca800ee50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:28:27.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:27.510+0000 7f9ca77fe640 1 -- 172.21.15.142:0/1854945191 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ca800cd00 con 0x7f9cb8102ae0 2024-09-06T14:28:27.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:27.510+0000 7f9cbce6c640 1 -- 172.21.15.142:0/1854945191 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9cb819ea30 con 0x7f9cb8102ae0 2024-09-06T14:28:27.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:27.510+0000 7f9ca77fe640 1 -- 172.21.15.142:0/1854945191 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7f9ca80035f0 con 0x7f9cb8102ae0 2024-09-06T14:28:27.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:27.510+0000 7f9ca77fe640 1 -- 172.21.15.142:0/1854945191 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ca80044a0 con 0x7f9cb8102ae0 2024-09-06T14:28:27.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:27.510+0000 7f9cbce6c640 1 -- 172.21.15.142:0/1854945191 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9cb81a52c0 con 0x7f9cb8102ae0 2024-09-06T14:28:27.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:27.511+0000 7f9cbce6c640 1 -- 172.21.15.142:0/1854945191 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9cb8102ec0 con 0x7f9cb8102ae0 2024-09-06T14:28:27.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:27.512+0000 7f9ca77fe640 1 -- 172.21.15.142:0/1854945191 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 22) ==== 99844+0+0 (secure 0 0 0) 0x7f9ca8004000 con 0x7f9cb8102ae0 2024-09-06T14:28:27.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:27.512+0000 7f9ca77fe640 1 --2- 172.21.15.142:0/1854945191 >> v2:172.21.15.142:6800/2944545612 conn(0x7f9ca0077620 0x7f9ca0079ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:28:27.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:27.512+0000 7f9ca77fe640 1 -- 172.21.15.142:0/1854945191 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(64..64 src has 1..64) ==== 6130+0+0 (secure 0 0 0) 0x7f9ca8099b70 con 0x7f9cb8102ae0 2024-09-06T14:28:27.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:27.512+0000 7f9cb5d9b640 1 --2- 172.21.15.142:0/1854945191 >> v2:172.21.15.142:6800/2944545612 conn(0x7f9ca0077620 0x7f9ca0079ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:28:27.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:27.513+0000 7f9cb5d9b640 1 --2- 172.21.15.142:0/1854945191 >> v2:172.21.15.142:6800/2944545612 conn(0x7f9ca0077620 0x7f9ca0079ae0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f9cb8107c20 tx=0x7f9cac0056d0 comp rx=0 tx=0).ready entity=mgr.14601 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:28:27.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:27.516+0000 7f9ca77fe640 1 -- 172.21.15.142:0/1854945191 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7f9ca8062c30 con 0x7f9cb8102ae0 2024-09-06T14:28:27.705 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:27.707+0000 7f9cbce6c640 1 -- 172.21.15.142:0/1854945191 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f9cb810ac50 con 0x7f9cb8102ae0 2024-09-06T14:28:27.705 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:27.708+0000 7f9ca77fe640 1 -- 172.21.15.142:0/1854945191 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+212 (secure 0 0 0) 0x7f9ca806b0e0 con 0x7f9cb8102ae0 2024-09-06T14:28:27.705 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:28:27.706 INFO:teuthology.orchestra.run.smithi142.stdout:{"status":"HEALTH_WARN","checks":{"MGR_MODULE_DEPENDENCY":{"severity":"HEALTH_WARN","summary":{"message":"Module 'volumes' has failed dependency: No module named 'ceph.fs'","count":1},"muted":false}},"mutes":[]} 2024-09-06T14:28:27.708 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:27.710+0000 7f9cbce6c640 1 -- 172.21.15.142:0/1854945191 >> v2:172.21.15.142:6800/2944545612 conn(0x7f9ca0077620 msgr2=0x7f9ca0079ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:28:27.708 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:27.710+0000 7f9cbce6c640 1 --2- 172.21.15.142:0/1854945191 >> v2:172.21.15.142:6800/2944545612 conn(0x7f9ca0077620 0x7f9ca0079ae0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f9cb8107c20 tx=0x7f9cac0056d0 comp rx=0 tx=0).stop 2024-09-06T14:28:27.708 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:27.710+0000 7f9cbce6c640 1 -- 172.21.15.142:0/1854945191 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9cb8102ae0 msgr2=0x7f9cb8198690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:28:27.708 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:27.710+0000 7f9cbce6c640 1 --2- 172.21.15.142:0/1854945191 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9cb8102ae0 0x7f9cb8198690 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f9ca8005fd0 tx=0x7f9ca800ee50 comp rx=0 tx=0).stop 2024-09-06T14:28:27.708 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:27.710+0000 7f9cbce6c640 1 -- 172.21.15.142:0/1854945191 shutdown_connections 2024-09-06T14:28:27.708 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:27.710+0000 7f9cbce6c640 1 --2- 172.21.15.142:0/1854945191 >> v2:172.21.15.142:6800/2944545612 conn(0x7f9ca0077620 0x7f9ca0079ae0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:27.708 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:27.710+0000 7f9cbce6c640 1 --2- 172.21.15.142:0/1854945191 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9cb8108e70 0x7f9cb819df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:27.708 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:27.710+0000 7f9cbce6c640 1 --2- 172.21.15.142:0/1854945191 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9cb81084f0 0x7f9cb8198bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:27.709 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:27.710+0000 7f9cbce6c640 1 --2- 172.21.15.142:0/1854945191 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9cb8102ae0 0x7f9cb8198690 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:27.709 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:27.710+0000 7f9cbce6c640 1 -- 172.21.15.142:0/1854945191 >> 172.21.15.142:0/1854945191 conn(0x7f9cb80fe550 msgr2=0x7f9cb8100940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:28:27.709 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:27.711+0000 7f9cbce6c640 1 -- 172.21.15.142:0/1854945191 shutdown_connections 2024-09-06T14:28:27.709 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:27.711+0000 7f9cbce6c640 1 -- 172.21.15.142:0/1854945191 wait complete. 2024-09-06T14:28:28.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:28:28 smithi162 ceph-mon[33787]: from='client.? 172.21.15.142:0/1854945191' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:28:28.345 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:28:28 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/1854945191' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:28:28.345 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:28:28 smithi142 ceph-mon[37372]: from='client.? 172.21.15.142:0/1854945191' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:28:29.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:28:29 smithi162 ceph-mon[33787]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-06T14:28:29.298 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:28:29 smithi162 ceph-mon[33787]: pgmap v111: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:28:29.459 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:28:29 smithi142 ceph-mon[29958]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-06T14:28:29.459 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:28:29 smithi142 ceph-mon[29958]: pgmap v111: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:28:29.460 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:28:29 smithi142 ceph-mon[37372]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-06T14:28:29.460 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:28:29 smithi142 ceph-mon[37372]: pgmap v111: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:28:31.373 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph health --format=json 2024-09-06T14:28:31.401 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:28:31 smithi142 ceph-mon[29958]: pgmap v112: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:28:31.402 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:28:31 smithi142 ceph-mon[37372]: pgmap v112: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:28:31.547 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:28:31 smithi162 ceph-mon[33787]: pgmap v112: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:28:31.636 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:28:32.110 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:32.081+0000 7fec1a0c6640 1 -- 172.21.15.142:0/3022928514 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fec14108e70 msgr2=0x7fec1410f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:28:32.110 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:32.081+0000 7fec1a0c6640 1 --2- 172.21.15.142:0/3022928514 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fec14108e70 0x7fec1410f670 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fec0c009be0 tx=0x7fec0c02a6c0 comp rx=0 tx=0).stop 2024-09-06T14:28:32.110 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:32.081+0000 7fec1a0c6640 1 -- 172.21.15.142:0/3022928514 shutdown_connections 2024-09-06T14:28:32.111 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:32.081+0000 7fec1a0c6640 1 --2- 172.21.15.142:0/3022928514 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fec14108e70 0x7fec1410f670 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:32.111 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:32.081+0000 7fec1a0c6640 1 --2- 172.21.15.142:0/3022928514 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fec141084f0 0x7fec14108930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:32.111 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:32.081+0000 7fec1a0c6640 1 --2- 172.21.15.142:0/3022928514 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fec14102ae0 0x7fec14102ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:32.111 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:32.081+0000 7fec1a0c6640 1 -- 172.21.15.142:0/3022928514 >> 172.21.15.142:0/3022928514 conn(0x7fec140fe530 msgr2=0x7fec14100950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:28:32.111 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:32.081+0000 7fec1a0c6640 1 -- 172.21.15.142:0/3022928514 shutdown_connections 2024-09-06T14:28:32.111 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:32.081+0000 7fec1a0c6640 1 -- 172.21.15.142:0/3022928514 wait complete. 2024-09-06T14:28:32.111 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:32.081+0000 7fec1a0c6640 1 Processor -- start 2024-09-06T14:28:32.111 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:32.082+0000 7fec1a0c6640 1 -- start start 2024-09-06T14:28:32.111 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:32.082+0000 7fec1a0c6640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fec14102ae0 0x7fec141986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:28:32.111 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:32.082+0000 7fec1a0c6640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fec141084f0 0x7fec14198c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:28:32.111 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:32.082+0000 7fec1a0c6640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fec14108e70 0x7fec1419dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:28:32.111 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:32.082+0000 7fec1a0c6640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fec14112240 con 0x7fec14108e70 2024-09-06T14:28:32.111 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:32.082+0000 7fec1a0c6640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fec141120c0 con 0x7fec141084f0 2024-09-06T14:28:32.111 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:32.082+0000 7fec1a0c6640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7fec141123c0 con 0x7fec14102ae0 2024-09-06T14:28:32.112 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:32.082+0000 7fec137fe640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fec14102ae0 0x7fec141986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:28:32.112 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:32.082+0000 7fec137fe640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fec14102ae0 0x7fec141986d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:38742/0 (socket says 172.21.15.142:38742) 2024-09-06T14:28:32.112 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:32.082+0000 7fec137fe640 1 -- 172.21.15.142:0/606261198 learned_addr learned my addr 172.21.15.142:0/606261198 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:28:32.112 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:32.082+0000 7fec13fff640 1 --2- 172.21.15.142:0/606261198 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fec14108e70 0x7fec1419dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:28:32.112 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:32.082+0000 7fec12ffd640 1 --2- 172.21.15.142:0/606261198 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fec141084f0 0x7fec14198c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:28:32.112 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:32.082+0000 7fec137fe640 1 -- 172.21.15.142:0/606261198 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fec141084f0 msgr2=0x7fec14198c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:28:32.112 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:32.082+0000 7fec137fe640 1 --2- 172.21.15.142:0/606261198 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fec141084f0 0x7fec14198c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:32.112 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:32.082+0000 7fec137fe640 1 -- 172.21.15.142:0/606261198 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fec14108e70 msgr2=0x7fec1419dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:28:32.112 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:32.082+0000 7fec137fe640 1 --2- 172.21.15.142:0/606261198 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fec14108e70 0x7fec1419dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:32.112 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:32.082+0000 7fec137fe640 1 -- 172.21.15.142:0/606261198 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec1419e780 con 0x7fec14102ae0 2024-09-06T14:28:32.112 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:32.082+0000 7fec12ffd640 1 --2- 172.21.15.142:0/606261198 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fec141084f0 0x7fec14198c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-06T14:28:32.112 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:32.082+0000 7fec13fff640 1 --2- 172.21.15.142:0/606261198 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fec14108e70 0x7fec1419dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-06T14:28:32.113 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:32.083+0000 7fec137fe640 1 --2- 172.21.15.142:0/606261198 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fec14102ae0 0x7fec141986d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fec0400e9c0 tx=0x7fec0400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:28:32.113 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:32.083+0000 7fec10ff9640 1 -- 172.21.15.142:0/606261198 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec0400cde0 con 0x7fec14102ae0 2024-09-06T14:28:32.113 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:32.083+0000 7fec1a0c6640 1 -- 172.21.15.142:0/606261198 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec1419ea70 con 0x7fec14102ae0 2024-09-06T14:28:32.113 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:32.083+0000 7fec10ff9640 1 -- 172.21.15.142:0/606261198 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7fec040035f0 con 0x7fec14102ae0 2024-09-06T14:28:32.113 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:32.083+0000 7fec10ff9640 1 -- 172.21.15.142:0/606261198 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec040044a0 con 0x7fec14102ae0 2024-09-06T14:28:32.113 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:32.083+0000 7fec1a0c6640 1 -- 172.21.15.142:0/606261198 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fec141a5300 con 0x7fec14102ae0 2024-09-06T14:28:32.113 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:32.084+0000 7fec1a0c6640 1 -- 172.21.15.142:0/606261198 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec14102ec0 con 0x7fec14102ae0 2024-09-06T14:28:32.113 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:32.084+0000 7fec10ff9640 1 -- 172.21.15.142:0/606261198 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 22) ==== 99844+0+0 (secure 0 0 0) 0x7fec04004000 con 0x7fec14102ae0 2024-09-06T14:28:32.113 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:32.085+0000 7fec10ff9640 1 --2- 172.21.15.142:0/606261198 >> v2:172.21.15.142:6800/2944545612 conn(0x7febfc077620 0x7febfc079ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:28:32.113 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:32.085+0000 7fec10ff9640 1 -- 172.21.15.142:0/606261198 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(64..64 src has 1..64) ==== 6130+0+0 (secure 0 0 0) 0x7fec04099d10 con 0x7fec14102ae0 2024-09-06T14:28:32.113 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:32.085+0000 7fec12ffd640 1 --2- 172.21.15.142:0/606261198 >> v2:172.21.15.142:6800/2944545612 conn(0x7febfc077620 0x7febfc079ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:28:32.113 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:32.085+0000 7fec12ffd640 1 --2- 172.21.15.142:0/606261198 >> v2:172.21.15.142:6800/2944545612 conn(0x7febfc077620 0x7febfc079ae0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fec14104d00 tx=0x7fec08008040 comp rx=0 tx=0).ready entity=mgr.14601 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:28:32.113 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:32.089+0000 7fec10ff9640 1 -- 172.21.15.142:0/606261198 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7fec04062dd0 con 0x7fec14102ae0 2024-09-06T14:28:32.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:32.279+0000 7fec1a0c6640 1 -- 172.21.15.142:0/606261198 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fec1410ac50 con 0x7fec14102ae0 2024-09-06T14:28:32.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:32.280+0000 7fec10ff9640 1 -- 172.21.15.142:0/606261198 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+212 (secure 0 0 0) 0x7fec0406b280 con 0x7fec14102ae0 2024-09-06T14:28:32.277 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:28:32.278 INFO:teuthology.orchestra.run.smithi142.stdout:{"status":"HEALTH_WARN","checks":{"MGR_MODULE_DEPENDENCY":{"severity":"HEALTH_WARN","summary":{"message":"Module 'volumes' has failed dependency: No module named 'ceph.fs'","count":1},"muted":false}},"mutes":[]} 2024-09-06T14:28:32.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:32.282+0000 7fec1a0c6640 1 -- 172.21.15.142:0/606261198 >> v2:172.21.15.142:6800/2944545612 conn(0x7febfc077620 msgr2=0x7febfc079ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:28:32.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:32.282+0000 7fec1a0c6640 1 --2- 172.21.15.142:0/606261198 >> v2:172.21.15.142:6800/2944545612 conn(0x7febfc077620 0x7febfc079ae0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fec14104d00 tx=0x7fec08008040 comp rx=0 tx=0).stop 2024-09-06T14:28:32.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:32.282+0000 7fec1a0c6640 1 -- 172.21.15.142:0/606261198 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fec14102ae0 msgr2=0x7fec141986d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:28:32.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:32.282+0000 7fec1a0c6640 1 --2- 172.21.15.142:0/606261198 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fec14102ae0 0x7fec141986d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fec0400e9c0 tx=0x7fec0400ee80 comp rx=0 tx=0).stop 2024-09-06T14:28:32.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:32.282+0000 7fec1a0c6640 1 -- 172.21.15.142:0/606261198 shutdown_connections 2024-09-06T14:28:32.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:32.282+0000 7fec1a0c6640 1 --2- 172.21.15.142:0/606261198 >> v2:172.21.15.142:6800/2944545612 conn(0x7febfc077620 0x7febfc079ae0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:32.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:32.282+0000 7fec1a0c6640 1 --2- 172.21.15.142:0/606261198 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fec14108e70 0x7fec1419dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:32.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:32.282+0000 7fec1a0c6640 1 --2- 172.21.15.142:0/606261198 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fec141084f0 0x7fec14198c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:32.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:32.282+0000 7fec1a0c6640 1 --2- 172.21.15.142:0/606261198 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fec14102ae0 0x7fec141986d0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:32.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:32.282+0000 7fec1a0c6640 1 -- 172.21.15.142:0/606261198 >> 172.21.15.142:0/606261198 conn(0x7fec140fe530 msgr2=0x7fec14100920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:28:32.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:32.282+0000 7fec1a0c6640 1 -- 172.21.15.142:0/606261198 shutdown_connections 2024-09-06T14:28:32.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:32.282+0000 7fec1a0c6640 1 -- 172.21.15.142:0/606261198 wait complete. 2024-09-06T14:28:33.459 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:28:33 smithi142 ceph-mon[29958]: pgmap v113: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:28:33.459 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:28:33 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/606261198' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:28:33.460 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:28:33 smithi142 ceph-mon[37372]: pgmap v113: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:28:33.460 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:28:33 smithi142 ceph-mon[37372]: from='client.? 172.21.15.142:0/606261198' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:28:33.547 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:28:33 smithi162 ceph-mon[33787]: pgmap v113: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:28:33.548 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:28:33 smithi162 ceph-mon[33787]: from='client.? 172.21.15.142:0/606261198' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:28:34.709 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:28:34 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: ::ffff:172.21.15.162 - - [06/Sep/2024:14:28:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-06T14:28:35.459 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:28:35 smithi142 ceph-mon[29958]: pgmap v114: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:28:35.459 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:28:35 smithi142 ceph-mon[37372]: pgmap v114: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:28:35.547 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:28:35 smithi162 ceph-mon[33787]: pgmap v114: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:28:35.957 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph health --format=json 2024-09-06T14:28:36.222 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:28:36.712 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:36.686+0000 7f254e8e6640 1 -- 172.21.15.142:0/425658183 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2548104320 msgr2=0x7f254810c810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:28:36.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:36.686+0000 7f254e8e6640 1 --2- 172.21.15.142:0/425658183 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2548104320 0x7f254810c810 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f2530009900 tx=0x7f253002a410 comp rx=0 tx=0).stop 2024-09-06T14:28:36.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:36.686+0000 7f254e8e6640 1 -- 172.21.15.142:0/425658183 shutdown_connections 2024-09-06T14:28:36.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:36.686+0000 7f254e8e6640 1 --2- 172.21.15.142:0/425658183 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f254810ce60 0x7f254810f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:36.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:36.686+0000 7f254e8e6640 1 --2- 172.21.15.142:0/425658183 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2548104320 0x7f254810c810 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:36.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:36.686+0000 7f254e8e6640 1 --2- 172.21.15.142:0/425658183 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2548103a00 0x7f2548103de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:36.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:36.686+0000 7f254e8e6640 1 -- 172.21.15.142:0/425658183 >> 172.21.15.142:0/425658183 conn(0x7f25480fd440 msgr2=0x7f25480ff860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:28:36.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:36.686+0000 7f254e8e6640 1 -- 172.21.15.142:0/425658183 shutdown_connections 2024-09-06T14:28:36.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:36.686+0000 7f254e8e6640 1 -- 172.21.15.142:0/425658183 wait complete. 2024-09-06T14:28:36.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:36.686+0000 7f254e8e6640 1 Processor -- start 2024-09-06T14:28:36.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:36.686+0000 7f254e8e6640 1 -- start start 2024-09-06T14:28:36.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:36.687+0000 7f254e8e6640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2548103a00 0x7f2548198b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:28:36.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:36.687+0000 7f254e8e6640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2548104320 0x7f2548199080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:28:36.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:36.687+0000 7f254e8e6640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f254810ce60 0x7f254819e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:28:36.714 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:36.687+0000 7f254e8e6640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f25481121e0 con 0x7f2548104320 2024-09-06T14:28:36.714 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:36.687+0000 7f254e8e6640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f2548112060 con 0x7f254810ce60 2024-09-06T14:28:36.714 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:36.687+0000 7f254e8e6640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7f2548112360 con 0x7f2548103a00 2024-09-06T14:28:36.716 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:36.687+0000 7f2547fff640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2548103a00 0x7f2548198b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:28:36.717 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:36.687+0000 7f25477fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2548104320 0x7f2548199080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:28:36.717 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:36.687+0000 7f2547fff640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2548103a00 0x7f2548198b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:38752/0 (socket says 172.21.15.142:38752) 2024-09-06T14:28:36.717 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:36.687+0000 7f2547fff640 1 -- 172.21.15.142:0/1477035453 learned_addr learned my addr 172.21.15.142:0/1477035453 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:28:36.717 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:36.687+0000 7f25477fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2548104320 0x7f2548199080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:39296/0 (socket says 172.21.15.142:39296) 2024-09-06T14:28:36.717 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:36.687+0000 7f2547fff640 1 -- 172.21.15.142:0/1477035453 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f254810ce60 msgr2=0x7f254819e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:28:36.717 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:36.687+0000 7f254ce83640 1 --2- 172.21.15.142:0/1477035453 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f254810ce60 0x7f254819e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:28:36.717 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:36.687+0000 7f2547fff640 1 --2- 172.21.15.142:0/1477035453 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f254810ce60 0x7f254819e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:36.717 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:36.687+0000 7f2547fff640 1 -- 172.21.15.142:0/1477035453 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2548104320 msgr2=0x7f2548199080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:28:36.717 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:36.687+0000 7f2547fff640 1 --2- 172.21.15.142:0/1477035453 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2548104320 0x7f2548199080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:36.717 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:36.687+0000 7f2547fff640 1 -- 172.21.15.142:0/1477035453 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f254819ebf0 con 0x7f2548103a00 2024-09-06T14:28:36.717 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:36.687+0000 7f25477fe640 1 --2- 172.21.15.142:0/1477035453 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2548104320 0x7f2548199080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-06T14:28:36.717 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:36.687+0000 7f2547fff640 1 --2- 172.21.15.142:0/1477035453 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2548103a00 0x7f2548198b40 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f253800b4a0 tx=0x7f253800b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:28:36.717 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:36.688+0000 7f25457fa640 1 -- 172.21.15.142:0/1477035453 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2538014070 con 0x7f2548103a00 2024-09-06T14:28:36.717 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:36.688+0000 7f254e8e6640 1 -- 172.21.15.142:0/1477035453 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f254819eee0 con 0x7f2548103a00 2024-09-06T14:28:36.718 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:36.688+0000 7f25457fa640 1 -- 172.21.15.142:0/1477035453 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7f2538010a90 con 0x7f2548103a00 2024-09-06T14:28:36.718 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:36.688+0000 7f25457fa640 1 -- 172.21.15.142:0/1477035453 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2538017b80 con 0x7f2548103a00 2024-09-06T14:28:36.718 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:36.688+0000 7f254e8e6640 1 -- 172.21.15.142:0/1477035453 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f25481a5770 con 0x7f2548103a00 2024-09-06T14:28:36.718 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:36.689+0000 7f254e8e6640 1 -- 172.21.15.142:0/1477035453 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2548103de0 con 0x7f2548103a00 2024-09-06T14:28:36.718 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:36.689+0000 7f25457fa640 1 -- 172.21.15.142:0/1477035453 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 22) ==== 99844+0+0 (secure 0 0 0) 0x7f2538017070 con 0x7f2548103a00 2024-09-06T14:28:36.718 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:36.690+0000 7f25457fa640 1 --2- 172.21.15.142:0/1477035453 >> v2:172.21.15.142:6800/2944545612 conn(0x7f2534077620 0x7f2534079ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:28:36.718 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:36.690+0000 7f25477fe640 1 --2- 172.21.15.142:0/1477035453 >> v2:172.21.15.142:6800/2944545612 conn(0x7f2534077620 0x7f2534079ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:28:36.718 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:36.690+0000 7f25457fa640 1 -- 172.21.15.142:0/1477035453 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(64..64 src has 1..64) ==== 6130+0+0 (secure 0 0 0) 0x7f2538098ec0 con 0x7f2548103a00 2024-09-06T14:28:36.718 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:36.690+0000 7f25477fe640 1 --2- 172.21.15.142:0/1477035453 >> v2:172.21.15.142:6800/2944545612 conn(0x7f2534077620 0x7f2534079ae0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f2530000c00 tx=0x7f25300318f0 comp rx=0 tx=0).ready entity=mgr.14601 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:28:36.718 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:36.694+0000 7f25457fa640 1 -- 172.21.15.142:0/1477035453 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7f25380660f0 con 0x7f2548103a00 2024-09-06T14:28:36.879 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:36.882+0000 7f254e8e6640 1 -- 172.21.15.142:0/1477035453 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f25481075e0 con 0x7f2548103a00 2024-09-06T14:28:36.880 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:36.882+0000 7f25457fa640 1 -- 172.21.15.142:0/1477035453 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+212 (secure 0 0 0) 0x7f253806ab10 con 0x7f2548103a00 2024-09-06T14:28:36.880 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:28:36.880 INFO:teuthology.orchestra.run.smithi142.stdout:{"status":"HEALTH_WARN","checks":{"MGR_MODULE_DEPENDENCY":{"severity":"HEALTH_WARN","summary":{"message":"Module 'volumes' has failed dependency: No module named 'ceph.fs'","count":1},"muted":false}},"mutes":[]} 2024-09-06T14:28:36.881 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:36.884+0000 7f254e8e6640 1 -- 172.21.15.142:0/1477035453 >> v2:172.21.15.142:6800/2944545612 conn(0x7f2534077620 msgr2=0x7f2534079ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:28:36.881 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:36.884+0000 7f254e8e6640 1 --2- 172.21.15.142:0/1477035453 >> v2:172.21.15.142:6800/2944545612 conn(0x7f2534077620 0x7f2534079ae0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f2530000c00 tx=0x7f25300318f0 comp rx=0 tx=0).stop 2024-09-06T14:28:36.882 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:36.884+0000 7f254e8e6640 1 -- 172.21.15.142:0/1477035453 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2548103a00 msgr2=0x7f2548198b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:28:36.882 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:36.884+0000 7f254e8e6640 1 --2- 172.21.15.142:0/1477035453 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2548103a00 0x7f2548198b40 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f253800b4a0 tx=0x7f253800b960 comp rx=0 tx=0).stop 2024-09-06T14:28:36.882 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:36.884+0000 7f254e8e6640 1 -- 172.21.15.142:0/1477035453 shutdown_connections 2024-09-06T14:28:36.882 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:36.884+0000 7f254e8e6640 1 --2- 172.21.15.142:0/1477035453 >> v2:172.21.15.142:6800/2944545612 conn(0x7f2534077620 0x7f2534079ae0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:36.882 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:36.884+0000 7f254e8e6640 1 --2- 172.21.15.142:0/1477035453 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f254810ce60 0x7f254819e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:36.882 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:36.884+0000 7f254e8e6640 1 --2- 172.21.15.142:0/1477035453 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2548104320 0x7f2548199080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:36.882 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:36.884+0000 7f254e8e6640 1 --2- 172.21.15.142:0/1477035453 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2548103a00 0x7f2548198b40 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:36.882 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:36.884+0000 7f254e8e6640 1 -- 172.21.15.142:0/1477035453 >> 172.21.15.142:0/1477035453 conn(0x7f25480fd440 msgr2=0x7f25481047b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:28:36.882 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:36.884+0000 7f254e8e6640 1 -- 172.21.15.142:0/1477035453 shutdown_connections 2024-09-06T14:28:36.882 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:36.884+0000 7f254e8e6640 1 -- 172.21.15.142:0/1477035453 wait complete. 2024-09-06T14:28:37.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:28:37 smithi142 ceph-mon[29958]: pgmap v115: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:28:37.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:28:37 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/1477035453' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:28:37.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:28:37 smithi142 ceph-mon[37372]: pgmap v115: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:28:37.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:28:37 smithi142 ceph-mon[37372]: from='client.? 172.21.15.142:0/1477035453' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:28:37.548 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:28:37 smithi162 ceph-mon[33787]: pgmap v115: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:28:37.548 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:28:37 smithi162 ceph-mon[33787]: from='client.? 172.21.15.142:0/1477035453' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:28:37.548 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 06 14:28:37 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-iscsi-iscsi-a[59853]: debug there is no tcmu-runner data available 2024-09-06T14:28:39.459 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:28:39 smithi142 ceph-mon[29958]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-06T14:28:39.459 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:28:39 smithi142 ceph-mon[29958]: pgmap v116: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:28:39.459 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:28:39 smithi142 ceph-mon[37372]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-06T14:28:39.460 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:28:39 smithi142 ceph-mon[37372]: pgmap v116: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:28:39.547 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:28:39 smithi162 ceph-mon[33787]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-06T14:28:39.548 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:28:39 smithi162 ceph-mon[33787]: pgmap v116: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:28:40.780 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph health --format=json 2024-09-06T14:28:41.042 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:28:41.320 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:28:41 smithi142 ceph-mon[29958]: pgmap v117: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:28:41.321 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:28:41 smithi142 ceph-mon[29958]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-06T14:28:41.321 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:28:41 smithi142 ceph-mon[37372]: pgmap v117: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:28:41.321 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:28:41 smithi142 ceph-mon[37372]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-06T14:28:41.547 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:28:41 smithi162 ceph-mon[33787]: pgmap v117: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:28:41.548 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:28:41 smithi162 ceph-mon[33787]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-06T14:28:41.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:41.535+0000 7fcae50da640 1 -- 172.21.15.142:0/1379255335 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcae0101560 msgr2=0x7fcae0101940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:28:41.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:41.535+0000 7fcae50da640 1 --2- 172.21.15.142:0/1379255335 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcae0101560 0x7fcae0101940 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fcad0009860 tx=0x7fcad002a3d0 comp rx=0 tx=0).stop 2024-09-06T14:28:41.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:41.535+0000 7fcae50da640 1 -- 172.21.15.142:0/1379255335 shutdown_connections 2024-09-06T14:28:41.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:41.535+0000 7fcae50da640 1 --2- 172.21.15.142:0/1379255335 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcae010f110 0x7fcae0111500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:41.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:41.535+0000 7fcae50da640 1 --2- 172.21.15.142:0/1379255335 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcae0101e80 0x7fcae010ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:41.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:41.535+0000 7fcae50da640 1 --2- 172.21.15.142:0/1379255335 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcae0101560 0x7fcae0101940 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:41.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:41.535+0000 7fcae50da640 1 -- 172.21.15.142:0/1379255335 >> 172.21.15.142:0/1379255335 conn(0x7fcae00fd430 msgr2=0x7fcae00ff850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:28:41.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:41.535+0000 7fcae50da640 1 -- 172.21.15.142:0/1379255335 shutdown_connections 2024-09-06T14:28:41.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:41.535+0000 7fcae50da640 1 -- 172.21.15.142:0/1379255335 wait complete. 2024-09-06T14:28:41.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:41.536+0000 7fcae50da640 1 Processor -- start 2024-09-06T14:28:41.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:41.536+0000 7fcae50da640 1 -- start start 2024-09-06T14:28:41.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:41.536+0000 7fcae50da640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcae0101560 0x7fcae019a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:28:41.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:41.537+0000 7fcae50da640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcae0101e80 0x7fcae019af10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:28:41.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:41.537+0000 7fcae50da640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcae010f110 0x7fcae01a02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:28:41.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:41.537+0000 7fcae50da640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fcae01143a0 con 0x7fcae0101560 2024-09-06T14:28:41.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:41.537+0000 7fcae50da640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fcae0114220 con 0x7fcae0101e80 2024-09-06T14:28:41.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:41.537+0000 7fcae50da640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7fcae0114520 con 0x7fcae010f110 2024-09-06T14:28:41.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:41.537+0000 7fcaded9d640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcae0101560 0x7fcae019a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:28:41.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:41.537+0000 7fcaded9d640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcae0101560 0x7fcae019a9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:50360/0 (socket says 172.21.15.142:50360) 2024-09-06T14:28:41.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:41.537+0000 7fcaded9d640 1 -- 172.21.15.142:0/4276596841 learned_addr learned my addr 172.21.15.142:0/4276596841 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:28:41.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:41.537+0000 7fcadf59e640 1 --2- 172.21.15.142:0/4276596841 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcae010f110 0x7fcae01a02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:28:41.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:41.537+0000 7fcade59c640 1 --2- 172.21.15.142:0/4276596841 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcae0101e80 0x7fcae019af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:28:41.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:41.537+0000 7fcadf59e640 1 -- 172.21.15.142:0/4276596841 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcae0101e80 msgr2=0x7fcae019af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:28:41.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:41.537+0000 7fcadf59e640 1 --2- 172.21.15.142:0/4276596841 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcae0101e80 0x7fcae019af10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:41.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:41.537+0000 7fcadf59e640 1 -- 172.21.15.142:0/4276596841 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcae0101560 msgr2=0x7fcae019a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:28:41.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:41.537+0000 7fcadf59e640 1 --2- 172.21.15.142:0/4276596841 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcae0101560 0x7fcae019a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:41.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:41.537+0000 7fcadf59e640 1 -- 172.21.15.142:0/4276596841 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcae01a0a80 con 0x7fcae010f110 2024-09-06T14:28:41.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:41.537+0000 7fcade59c640 1 --2- 172.21.15.142:0/4276596841 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcae0101e80 0x7fcae019af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-06T14:28:41.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:41.537+0000 7fcaded9d640 1 --2- 172.21.15.142:0/4276596841 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcae0101560 0x7fcae019a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-06T14:28:41.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:41.537+0000 7fcadf59e640 1 --2- 172.21.15.142:0/4276596841 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcae010f110 0x7fcae01a02b0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fcad800be20 tx=0x7fcad800bf10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:28:41.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:41.537+0000 7fcac7fff640 1 -- 172.21.15.142:0/4276596841 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcad8019070 con 0x7fcae010f110 2024-09-06T14:28:41.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:41.537+0000 7fcac7fff640 1 -- 172.21.15.142:0/4276596841 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7fcad80148c0 con 0x7fcae010f110 2024-09-06T14:28:41.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:41.537+0000 7fcae50da640 1 -- 172.21.15.142:0/4276596841 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcae01a0d70 con 0x7fcae010f110 2024-09-06T14:28:41.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:41.537+0000 7fcac7fff640 1 -- 172.21.15.142:0/4276596841 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcad80126e0 con 0x7fcae010f110 2024-09-06T14:28:41.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:41.537+0000 7fcae50da640 1 -- 172.21.15.142:0/4276596841 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fcae01a7430 con 0x7fcae010f110 2024-09-06T14:28:41.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:41.538+0000 7fcae50da640 1 -- 172.21.15.142:0/4276596841 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcae0101940 con 0x7fcae010f110 2024-09-06T14:28:41.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:41.539+0000 7fcac7fff640 1 -- 172.21.15.142:0/4276596841 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 22) ==== 99844+0+0 (secure 0 0 0) 0x7fcad801c070 con 0x7fcae010f110 2024-09-06T14:28:41.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:41.539+0000 7fcac7fff640 1 --2- 172.21.15.142:0/4276596841 >> v2:172.21.15.142:6800/2944545612 conn(0x7fcac8077620 0x7fcac8079ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:28:41.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:41.540+0000 7fcac7fff640 1 -- 172.21.15.142:0/4276596841 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(64..64 src has 1..64) ==== 6130+0+0 (secure 0 0 0) 0x7fcad8099130 con 0x7fcae010f110 2024-09-06T14:28:41.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:41.540+0000 7fcaded9d640 1 --2- 172.21.15.142:0/4276596841 >> v2:172.21.15.142:6800/2944545612 conn(0x7fcac8077620 0x7fcac8079ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:28:41.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:41.540+0000 7fcaded9d640 1 --2- 172.21.15.142:0/4276596841 >> v2:172.21.15.142:6800/2944545612 conn(0x7fcac8077620 0x7fcac8079ae0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fcad00095f0 tx=0x7fcad00058b0 comp rx=0 tx=0).ready entity=mgr.14601 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:28:41.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:41.544+0000 7fcac7fff640 1 -- 172.21.15.142:0/4276596841 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7fcad8066360 con 0x7fcae010f110 2024-09-06T14:28:41.731 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:41.734+0000 7fcae50da640 1 -- 172.21.15.142:0/4276596841 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fcae01a76c0 con 0x7fcae010f110 2024-09-06T14:28:41.732 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:41.734+0000 7fcac7fff640 1 -- 172.21.15.142:0/4276596841 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+212 (secure 0 0 0) 0x7fcad806ad80 con 0x7fcae010f110 2024-09-06T14:28:41.732 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:28:41.732 INFO:teuthology.orchestra.run.smithi142.stdout:{"status":"HEALTH_WARN","checks":{"MGR_MODULE_DEPENDENCY":{"severity":"HEALTH_WARN","summary":{"message":"Module 'volumes' has failed dependency: No module named 'ceph.fs'","count":1},"muted":false}},"mutes":[]} 2024-09-06T14:28:41.734 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:41.736+0000 7fcae50da640 1 -- 172.21.15.142:0/4276596841 >> v2:172.21.15.142:6800/2944545612 conn(0x7fcac8077620 msgr2=0x7fcac8079ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:28:41.734 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:41.736+0000 7fcae50da640 1 --2- 172.21.15.142:0/4276596841 >> v2:172.21.15.142:6800/2944545612 conn(0x7fcac8077620 0x7fcac8079ae0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fcad00095f0 tx=0x7fcad00058b0 comp rx=0 tx=0).stop 2024-09-06T14:28:41.734 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:41.736+0000 7fcae50da640 1 -- 172.21.15.142:0/4276596841 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcae010f110 msgr2=0x7fcae01a02b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:28:41.734 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:41.736+0000 7fcae50da640 1 --2- 172.21.15.142:0/4276596841 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcae010f110 0x7fcae01a02b0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fcad800be20 tx=0x7fcad800bf10 comp rx=0 tx=0).stop 2024-09-06T14:28:41.734 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:41.736+0000 7fcae50da640 1 -- 172.21.15.142:0/4276596841 shutdown_connections 2024-09-06T14:28:41.734 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:41.736+0000 7fcae50da640 1 --2- 172.21.15.142:0/4276596841 >> v2:172.21.15.142:6800/2944545612 conn(0x7fcac8077620 0x7fcac8079ae0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:41.735 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:41.736+0000 7fcae50da640 1 --2- 172.21.15.142:0/4276596841 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcae010f110 0x7fcae01a02b0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:41.735 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:41.736+0000 7fcae50da640 1 --2- 172.21.15.142:0/4276596841 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcae0101e80 0x7fcae019af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:41.735 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:41.736+0000 7fcae50da640 1 --2- 172.21.15.142:0/4276596841 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcae0101560 0x7fcae019a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:41.735 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:41.736+0000 7fcae50da640 1 -- 172.21.15.142:0/4276596841 >> 172.21.15.142:0/4276596841 conn(0x7fcae00fd430 msgr2=0x7fcae010f730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:28:41.735 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:41.737+0000 7fcae50da640 1 -- 172.21.15.142:0/4276596841 shutdown_connections 2024-09-06T14:28:41.735 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:41.737+0000 7fcae50da640 1 -- 172.21.15.142:0/4276596841 wait complete. 2024-09-06T14:28:42.356 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:28:42 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/4276596841' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:28:42.357 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:28:42 smithi142 ceph-mon[37372]: from='client.? 172.21.15.142:0/4276596841' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:28:42.547 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:28:42 smithi162 ceph-mon[33787]: from='client.? 172.21.15.142:0/4276596841' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:28:43.459 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:28:43 smithi142 ceph-mon[29958]: pgmap v118: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:28:43.459 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:28:43 smithi142 ceph-mon[37372]: pgmap v118: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:28:43.548 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:28:43 smithi162 ceph-mon[33787]: pgmap v118: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:28:44.709 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:28:44 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: ::ffff:172.21.15.162 - - [06/Sep/2024:14:28:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-06T14:28:45.350 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:28:45 smithi162 ceph-mon[33787]: pgmap v119: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:28:45.384 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph health --format=json 2024-09-06T14:28:45.412 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:28:45 smithi142 ceph-mon[29958]: pgmap v119: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:28:45.412 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:28:45 smithi142 ceph-mon[37372]: pgmap v119: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:28:45.645 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:28:46.126 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:46.114+0000 7fac7c2cb640 1 -- 172.21.15.142:0/3492781375 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fac741084e0 msgr2=0x7fac74108920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:28:46.126 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:46.114+0000 7fac7c2cb640 1 --2- 172.21.15.142:0/3492781375 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fac741084e0 0x7fac74108920 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fac68009860 tx=0x7fac6802a340 comp rx=0 tx=0).stop 2024-09-06T14:28:46.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:46.114+0000 7fac7c2cb640 1 -- 172.21.15.142:0/3492781375 shutdown_connections 2024-09-06T14:28:46.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:46.114+0000 7fac7c2cb640 1 --2- 172.21.15.142:0/3492781375 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fac74108e60 0x7fac7410f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:46.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:46.114+0000 7fac7c2cb640 1 --2- 172.21.15.142:0/3492781375 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fac741084e0 0x7fac74108920 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:46.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:46.114+0000 7fac7c2cb640 1 --2- 172.21.15.142:0/3492781375 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fac74102ad0 0x7fac74102eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:46.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:46.114+0000 7fac7c2cb640 1 -- 172.21.15.142:0/3492781375 >> 172.21.15.142:0/3492781375 conn(0x7fac740fe580 msgr2=0x7fac741009a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:28:46.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:46.114+0000 7fac7c2cb640 1 -- 172.21.15.142:0/3492781375 shutdown_connections 2024-09-06T14:28:46.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:46.114+0000 7fac7c2cb640 1 -- 172.21.15.142:0/3492781375 wait complete. 2024-09-06T14:28:46.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:46.115+0000 7fac7c2cb640 1 Processor -- start 2024-09-06T14:28:46.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:46.115+0000 7fac7c2cb640 1 -- start start 2024-09-06T14:28:46.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:46.115+0000 7fac7c2cb640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fac74102ad0 0x7fac741986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:28:46.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:46.115+0000 7fac7c2cb640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fac741084e0 0x7fac74198c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:28:46.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:46.115+0000 7fac7c2cb640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fac74108e60 0x7fac7419dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:28:46.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:46.115+0000 7fac7c2cb640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fac74112230 con 0x7fac74108e60 2024-09-06T14:28:46.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:46.115+0000 7fac7c2cb640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fac741120b0 con 0x7fac741084e0 2024-09-06T14:28:46.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:46.115+0000 7fac7c2cb640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7fac741123b0 con 0x7fac74102ad0 2024-09-06T14:28:46.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:46.115+0000 7fac7a067640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fac74102ad0 0x7fac741986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:28:46.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:46.115+0000 7fac7a067640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fac74102ad0 0x7fac741986e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:54020/0 (socket says 172.21.15.142:54020) 2024-09-06T14:28:46.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:46.115+0000 7fac7a067640 1 -- 172.21.15.142:0/2558221870 learned_addr learned my addr 172.21.15.142:0/2558221870 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:28:46.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:46.115+0000 7fac7a868640 1 --2- 172.21.15.142:0/2558221870 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fac74108e60 0x7fac7419dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:28:46.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:46.115+0000 7fac79866640 1 --2- 172.21.15.142:0/2558221870 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fac741084e0 0x7fac74198c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:28:46.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:46.115+0000 7fac7a067640 1 -- 172.21.15.142:0/2558221870 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fac741084e0 msgr2=0x7fac74198c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:28:46.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:46.115+0000 7fac7a067640 1 --2- 172.21.15.142:0/2558221870 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fac741084e0 0x7fac74198c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:46.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:46.115+0000 7fac7a067640 1 -- 172.21.15.142:0/2558221870 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fac74108e60 msgr2=0x7fac7419dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:28:46.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:46.115+0000 7fac7a067640 1 --2- 172.21.15.142:0/2558221870 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fac74108e60 0x7fac7419dfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:46.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:46.115+0000 7fac7a067640 1 -- 172.21.15.142:0/2558221870 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac7419e790 con 0x7fac74102ad0 2024-09-06T14:28:46.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:46.115+0000 7fac79866640 1 --2- 172.21.15.142:0/2558221870 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fac741084e0 0x7fac74198c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-06T14:28:46.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:46.116+0000 7fac7a067640 1 --2- 172.21.15.142:0/2558221870 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fac74102ad0 0x7fac741986e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fac6c00e9f0 tx=0x7fac6c00edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:28:46.129 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:46.116+0000 7fac737fe640 1 -- 172.21.15.142:0/2558221870 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac6c00cd80 con 0x7fac74102ad0 2024-09-06T14:28:46.129 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:46.116+0000 7fac737fe640 1 -- 172.21.15.142:0/2558221870 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7fac6c0054a0 con 0x7fac74102ad0 2024-09-06T14:28:46.129 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:46.116+0000 7fac7c2cb640 1 -- 172.21.15.142:0/2558221870 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac7419ea80 con 0x7fac74102ad0 2024-09-06T14:28:46.129 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:46.116+0000 7fac737fe640 1 -- 172.21.15.142:0/2558221870 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac6c010430 con 0x7fac74102ad0 2024-09-06T14:28:46.129 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:46.116+0000 7fac7c2cb640 1 -- 172.21.15.142:0/2558221870 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fac741a5140 con 0x7fac74102ad0 2024-09-06T14:28:46.129 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:46.117+0000 7fac7c2cb640 1 -- 172.21.15.142:0/2558221870 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac74102eb0 con 0x7fac74102ad0 2024-09-06T14:28:46.129 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:46.118+0000 7fac737fe640 1 -- 172.21.15.142:0/2558221870 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 22) ==== 99844+0+0 (secure 0 0 0) 0x7fac6c005020 con 0x7fac74102ad0 2024-09-06T14:28:46.129 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:46.118+0000 7fac737fe640 1 --2- 172.21.15.142:0/2558221870 >> v2:172.21.15.142:6800/2944545612 conn(0x7fac5c077620 0x7fac5c079ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:28:46.129 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:46.118+0000 7fac79866640 1 --2- 172.21.15.142:0/2558221870 >> v2:172.21.15.142:6800/2944545612 conn(0x7fac5c077620 0x7fac5c079ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:28:46.129 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:46.118+0000 7fac737fe640 1 -- 172.21.15.142:0/2558221870 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(64..64 src has 1..64) ==== 6130+0+0 (secure 0 0 0) 0x7fac6c099d60 con 0x7fac74102ad0 2024-09-06T14:28:46.129 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:46.118+0000 7fac79866640 1 --2- 172.21.15.142:0/2558221870 >> v2:172.21.15.142:6800/2944545612 conn(0x7fac5c077620 0x7fac5c079ae0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fac68005b60 tx=0x7fac68005a90 comp rx=0 tx=0).ready entity=mgr.14601 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:28:46.129 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:46.122+0000 7fac737fe640 1 -- 172.21.15.142:0/2558221870 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7fac6c062f60 con 0x7fac74102ad0 2024-09-06T14:28:46.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:46.310+0000 7fac7c2cb640 1 -- 172.21.15.142:0/2558221870 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fac7410ac40 con 0x7fac74102ad0 2024-09-06T14:28:46.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:46.310+0000 7fac737fe640 1 -- 172.21.15.142:0/2558221870 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+212 (secure 0 0 0) 0x7fac6c06b410 con 0x7fac74102ad0 2024-09-06T14:28:46.308 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:28:46.308 INFO:teuthology.orchestra.run.smithi142.stdout:{"status":"HEALTH_WARN","checks":{"MGR_MODULE_DEPENDENCY":{"severity":"HEALTH_WARN","summary":{"message":"Module 'volumes' has failed dependency: No module named 'ceph.fs'","count":1},"muted":false}},"mutes":[]} 2024-09-06T14:28:46.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:46.313+0000 7fac7c2cb640 1 -- 172.21.15.142:0/2558221870 >> v2:172.21.15.142:6800/2944545612 conn(0x7fac5c077620 msgr2=0x7fac5c079ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:28:46.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:46.313+0000 7fac7c2cb640 1 --2- 172.21.15.142:0/2558221870 >> v2:172.21.15.142:6800/2944545612 conn(0x7fac5c077620 0x7fac5c079ae0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fac68005b60 tx=0x7fac68005a90 comp rx=0 tx=0).stop 2024-09-06T14:28:46.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:46.313+0000 7fac7c2cb640 1 -- 172.21.15.142:0/2558221870 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fac74102ad0 msgr2=0x7fac741986e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:28:46.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:46.313+0000 7fac7c2cb640 1 --2- 172.21.15.142:0/2558221870 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fac74102ad0 0x7fac741986e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fac6c00e9f0 tx=0x7fac6c00edf0 comp rx=0 tx=0).stop 2024-09-06T14:28:46.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:46.313+0000 7fac7c2cb640 1 -- 172.21.15.142:0/2558221870 shutdown_connections 2024-09-06T14:28:46.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:46.313+0000 7fac7c2cb640 1 --2- 172.21.15.142:0/2558221870 >> v2:172.21.15.142:6800/2944545612 conn(0x7fac5c077620 0x7fac5c079ae0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:46.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:46.313+0000 7fac7c2cb640 1 --2- 172.21.15.142:0/2558221870 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fac74108e60 0x7fac7419dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:46.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:46.313+0000 7fac7c2cb640 1 --2- 172.21.15.142:0/2558221870 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fac741084e0 0x7fac74198c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:46.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:46.313+0000 7fac7c2cb640 1 --2- 172.21.15.142:0/2558221870 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fac74102ad0 0x7fac741986e0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:46.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:46.313+0000 7fac7c2cb640 1 -- 172.21.15.142:0/2558221870 >> 172.21.15.142:0/2558221870 conn(0x7fac740fe580 msgr2=0x7fac74100970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:28:46.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:46.313+0000 7fac7c2cb640 1 -- 172.21.15.142:0/2558221870 shutdown_connections 2024-09-06T14:28:46.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:46.313+0000 7fac7c2cb640 1 -- 172.21.15.142:0/2558221870 wait complete. 2024-09-06T14:28:47.459 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:28:47 smithi142 ceph-mon[29958]: pgmap v120: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:28:47.459 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:28:47 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/2558221870' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:28:47.460 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:28:47 smithi142 ceph-mon[37372]: pgmap v120: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:28:47.460 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:28:47 smithi142 ceph-mon[37372]: from='client.? 172.21.15.142:0/2558221870' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:28:47.547 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:28:47 smithi162 ceph-mon[33787]: pgmap v120: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:28:47.548 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:28:47 smithi162 ceph-mon[33787]: from='client.? 172.21.15.142:0/2558221870' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:28:47.548 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 06 14:28:47 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-iscsi-iscsi-a[59853]: debug there is no tcmu-runner data available 2024-09-06T14:28:49.459 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:28:49 smithi142 ceph-mon[29958]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-06T14:28:49.459 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:28:49 smithi142 ceph-mon[29958]: pgmap v121: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:28:49.459 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:28:49 smithi142 ceph-mon[37372]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-06T14:28:49.460 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:28:49 smithi142 ceph-mon[37372]: pgmap v121: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:28:49.547 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:28:49 smithi162 ceph-mon[33787]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-06T14:28:49.548 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:28:49 smithi162 ceph-mon[33787]: pgmap v121: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:28:50.037 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph health --format=json 2024-09-06T14:28:50.295 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:28:50.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:50.798+0000 7f152c546640 1 -- 172.21.15.142:0/3303649232 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f15241073d0 msgr2=0x7f1524107810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:28:50.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:50.798+0000 7f152c546640 1 --2- 172.21.15.142:0/3303649232 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f15241073d0 0x7f1524107810 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f1514009900 tx=0x7f151402a410 comp rx=0 tx=0).stop 2024-09-06T14:28:50.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:50.798+0000 7f152c546640 1 -- 172.21.15.142:0/3303649232 shutdown_connections 2024-09-06T14:28:50.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:50.798+0000 7f152c546640 1 --2- 172.21.15.142:0/3303649232 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1524107d50 0x7f152410e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:50.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:50.798+0000 7f152c546640 1 --2- 172.21.15.142:0/3303649232 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f15241073d0 0x7f1524107810 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:50.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:50.798+0000 7f152c546640 1 --2- 172.21.15.142:0/3303649232 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f15241019c0 0x7f1524101da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:50.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:50.798+0000 7f152c546640 1 -- 172.21.15.142:0/3303649232 >> 172.21.15.142:0/3303649232 conn(0x7f15240fd430 msgr2=0x7f15240ff850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:28:50.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:50.798+0000 7f152c546640 1 -- 172.21.15.142:0/3303649232 shutdown_connections 2024-09-06T14:28:50.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:50.798+0000 7f152c546640 1 -- 172.21.15.142:0/3303649232 wait complete. 2024-09-06T14:28:50.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:50.798+0000 7f152c546640 1 Processor -- start 2024-09-06T14:28:50.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:50.798+0000 7f152c546640 1 -- start start 2024-09-06T14:28:50.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:50.799+0000 7f152c546640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f15241019c0 0x7f1524199490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:28:50.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:50.799+0000 7f152c546640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f15241073d0 0x7f15241999d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:28:50.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:50.799+0000 7f152c546640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1524107d50 0x7f15241a4510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:28:50.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:50.799+0000 7f152c546640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f1524111120 con 0x7f15241019c0 2024-09-06T14:28:50.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:50.799+0000 7f152c546640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1524110fa0 con 0x7f15241073d0 2024-09-06T14:28:50.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:50.799+0000 7f152c546640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7f15241112a0 con 0x7f1524107d50 2024-09-06T14:28:50.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:50.799+0000 7f152a2e2640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f15241019c0 0x7f1524199490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:28:50.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:50.799+0000 7f152a2e2640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f15241019c0 0x7f1524199490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:38806/0 (socket says 172.21.15.142:38806) 2024-09-06T14:28:50.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:50.799+0000 7f152a2e2640 1 -- 172.21.15.142:0/113978840 learned_addr learned my addr 172.21.15.142:0/113978840 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:28:50.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:50.799+0000 7f152aae3640 1 --2- 172.21.15.142:0/113978840 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1524107d50 0x7f15241a4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:28:50.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:50.799+0000 7f1529ae1640 1 --2- 172.21.15.142:0/113978840 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f15241073d0 0x7f15241999d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:28:50.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:50.799+0000 7f152a2e2640 1 -- 172.21.15.142:0/113978840 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1524107d50 msgr2=0x7f15241a4510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:28:50.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:50.799+0000 7f152a2e2640 1 --2- 172.21.15.142:0/113978840 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1524107d50 0x7f15241a4510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:50.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:50.799+0000 7f152a2e2640 1 -- 172.21.15.142:0/113978840 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f15241073d0 msgr2=0x7f15241999d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:28:50.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:50.799+0000 7f152a2e2640 1 --2- 172.21.15.142:0/113978840 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f15241073d0 0x7f15241999d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:50.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:50.799+0000 7f152a2e2640 1 -- 172.21.15.142:0/113978840 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15241a4a50 con 0x7f15241019c0 2024-09-06T14:28:50.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:50.799+0000 7f152aae3640 1 --2- 172.21.15.142:0/113978840 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1524107d50 0x7f15241a4510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-06T14:28:50.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:50.799+0000 7f152a2e2640 1 --2- 172.21.15.142:0/113978840 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f15241019c0 0x7f1524199490 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f152000ea60 tx=0x7f152000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:28:50.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:50.799+0000 7f151b7fe640 1 -- 172.21.15.142:0/113978840 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f152000cde0 con 0x7f15241019c0 2024-09-06T14:28:50.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:50.799+0000 7f151b7fe640 1 -- 172.21.15.142:0/113978840 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7f15200035f0 con 0x7f15241019c0 2024-09-06T14:28:50.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:50.799+0000 7f152c546640 1 -- 172.21.15.142:0/113978840 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15241a4c20 con 0x7f15241019c0 2024-09-06T14:28:50.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:50.799+0000 7f151b7fe640 1 -- 172.21.15.142:0/113978840 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15200044a0 con 0x7f15241019c0 2024-09-06T14:28:50.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:50.799+0000 7f152c546640 1 -- 172.21.15.142:0/113978840 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15241a50d0 con 0x7f15241019c0 2024-09-06T14:28:50.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:50.800+0000 7f152c546640 1 -- 172.21.15.142:0/113978840 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1524101da0 con 0x7f15241019c0 2024-09-06T14:28:50.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:50.801+0000 7f151b7fe640 1 -- 172.21.15.142:0/113978840 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 22) ==== 99844+0+0 (secure 0 0 0) 0x7f1520004000 con 0x7f15241019c0 2024-09-06T14:28:50.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:50.801+0000 7f151b7fe640 1 --2- 172.21.15.142:0/113978840 >> v2:172.21.15.142:6800/2944545612 conn(0x7f1510077620 0x7f1510079ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:28:50.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:50.802+0000 7f1529ae1640 1 --2- 172.21.15.142:0/113978840 >> v2:172.21.15.142:6800/2944545612 conn(0x7f1510077620 0x7f1510079ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:28:50.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:50.802+0000 7f151b7fe640 1 -- 172.21.15.142:0/113978840 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 6130+0+0 (secure 0 0 0) 0x7f1520099c30 con 0x7f15241019c0 2024-09-06T14:28:50.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:50.802+0000 7f1529ae1640 1 --2- 172.21.15.142:0/113978840 >> v2:172.21.15.142:6800/2944545612 conn(0x7f1510077620 0x7f1510079ae0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f1514031440 tx=0x7f1514005b50 comp rx=0 tx=0).ready entity=mgr.14601 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:28:50.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:50.805+0000 7f151b7fe640 1 -- 172.21.15.142:0/113978840 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7f1520066e60 con 0x7f15241019c0 2024-09-06T14:28:50.991 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:50.993+0000 7f152c546640 1 -- 172.21.15.142:0/113978840 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f15241095a0 con 0x7f15241019c0 2024-09-06T14:28:50.991 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:50.993+0000 7f151b7fe640 1 -- 172.21.15.142:0/113978840 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+212 (secure 0 0 0) 0x7f152006b1a0 con 0x7f15241019c0 2024-09-06T14:28:50.991 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:28:50.991 INFO:teuthology.orchestra.run.smithi142.stdout:{"status":"HEALTH_WARN","checks":{"MGR_MODULE_DEPENDENCY":{"severity":"HEALTH_WARN","summary":{"message":"Module 'volumes' has failed dependency: No module named 'ceph.fs'","count":1},"muted":false}},"mutes":[]} 2024-09-06T14:28:50.994 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:50.996+0000 7f152c546640 1 -- 172.21.15.142:0/113978840 >> v2:172.21.15.142:6800/2944545612 conn(0x7f1510077620 msgr2=0x7f1510079ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:28:50.994 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:50.996+0000 7f152c546640 1 --2- 172.21.15.142:0/113978840 >> v2:172.21.15.142:6800/2944545612 conn(0x7f1510077620 0x7f1510079ae0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f1514031440 tx=0x7f1514005b50 comp rx=0 tx=0).stop 2024-09-06T14:28:50.994 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:50.996+0000 7f152c546640 1 -- 172.21.15.142:0/113978840 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f15241019c0 msgr2=0x7f1524199490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:28:50.994 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:50.996+0000 7f152c546640 1 --2- 172.21.15.142:0/113978840 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f15241019c0 0x7f1524199490 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f152000ea60 tx=0x7f152000ef20 comp rx=0 tx=0).stop 2024-09-06T14:28:50.994 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:50.996+0000 7f152c546640 1 -- 172.21.15.142:0/113978840 shutdown_connections 2024-09-06T14:28:50.994 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:50.996+0000 7f152c546640 1 --2- 172.21.15.142:0/113978840 >> v2:172.21.15.142:6800/2944545612 conn(0x7f1510077620 0x7f1510079ae0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:50.994 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:50.996+0000 7f152c546640 1 --2- 172.21.15.142:0/113978840 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1524107d50 0x7f15241a4510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:50.994 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:50.996+0000 7f152c546640 1 --2- 172.21.15.142:0/113978840 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f15241073d0 0x7f15241999d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:50.994 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:50.996+0000 7f152c546640 1 --2- 172.21.15.142:0/113978840 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f15241019c0 0x7f1524199490 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:50.994 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:50.996+0000 7f152c546640 1 -- 172.21.15.142:0/113978840 >> 172.21.15.142:0/113978840 conn(0x7f15240fd430 msgr2=0x7f15240ff820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:28:50.994 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:50.996+0000 7f152c546640 1 -- 172.21.15.142:0/113978840 shutdown_connections 2024-09-06T14:28:50.994 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:50.996+0000 7f152c546640 1 -- 172.21.15.142:0/113978840 wait complete. 2024-09-06T14:28:51.451 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:28:51 smithi142 ceph-mon[29958]: pgmap v122: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:28:51.451 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:28:51 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/113978840' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:28:51.451 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:28:51 smithi142 ceph-mon[37372]: pgmap v122: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:28:51.451 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:28:51 smithi142 ceph-mon[37372]: from='client.? 172.21.15.142:0/113978840' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:28:51.547 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:28:51 smithi162 ceph-mon[33787]: pgmap v122: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:28:51.548 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:28:51 smithi162 ceph-mon[33787]: from='client.? 172.21.15.142:0/113978840' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:28:53.459 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:28:53 smithi142 ceph-mon[29958]: pgmap v123: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:28:53.459 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:28:53 smithi142 ceph-mon[37372]: pgmap v123: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:28:53.548 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:28:53 smithi162 ceph-mon[33787]: pgmap v123: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:28:54.665 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph health --format=json 2024-09-06T14:28:54.709 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:28:54 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: ::ffff:172.21.15.162 - - [06/Sep/2024:14:28:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-06T14:28:54.913 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:28:55.315 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:28:55 smithi142 ceph-mon[29958]: pgmap v124: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:28:55.315 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:28:55 smithi142 ceph-mon[37372]: pgmap v124: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:28:55.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:55.396+0000 7f5e7a75b640 1 -- 172.21.15.142:0/3057520777 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5e74102ae0 msgr2=0x7f5e74102ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:28:55.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:55.396+0000 7f5e7a75b640 1 --2- 172.21.15.142:0/3057520777 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5e74102ae0 0x7f5e74102ec0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f5e64009900 tx=0x7f5e6402a410 comp rx=0 tx=0).stop 2024-09-06T14:28:55.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:55.396+0000 7f5e7a75b640 1 -- 172.21.15.142:0/3057520777 shutdown_connections 2024-09-06T14:28:55.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:55.396+0000 7f5e7a75b640 1 --2- 172.21.15.142:0/3057520777 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5e74108e70 0x7f5e7410f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:55.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:55.396+0000 7f5e7a75b640 1 --2- 172.21.15.142:0/3057520777 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e741084f0 0x7f5e74108930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:55.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:55.396+0000 7f5e7a75b640 1 --2- 172.21.15.142:0/3057520777 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5e74102ae0 0x7f5e74102ec0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:55.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:55.396+0000 7f5e7a75b640 1 -- 172.21.15.142:0/3057520777 >> 172.21.15.142:0/3057520777 conn(0x7f5e740fe530 msgr2=0x7f5e74100950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:28:55.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:55.396+0000 7f5e7a75b640 1 -- 172.21.15.142:0/3057520777 shutdown_connections 2024-09-06T14:28:55.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:55.396+0000 7f5e7a75b640 1 -- 172.21.15.142:0/3057520777 wait complete. 2024-09-06T14:28:55.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:55.397+0000 7f5e7a75b640 1 Processor -- start 2024-09-06T14:28:55.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:55.397+0000 7f5e7a75b640 1 -- start start 2024-09-06T14:28:55.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:55.397+0000 7f5e7a75b640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5e74102ae0 0x7f5e741986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:28:55.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:55.397+0000 7f5e7a75b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e741084f0 0x7f5e74198c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:28:55.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:55.397+0000 7f5e7a75b640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5e74108e70 0x7f5e7419dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:28:55.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:55.397+0000 7f5e7a75b640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f5e74112290 con 0x7f5e74102ae0 2024-09-06T14:28:55.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:55.397+0000 7f5e7a75b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5e74112110 con 0x7f5e741084f0 2024-09-06T14:28:55.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:55.397+0000 7f5e7a75b640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7f5e74112410 con 0x7f5e74108e70 2024-09-06T14:28:55.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:55.397+0000 7f5e73fff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5e74102ae0 0x7f5e741986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:28:55.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:55.397+0000 7f5e78cf8640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5e74108e70 0x7f5e7419dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:28:55.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:55.397+0000 7f5e73fff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5e74102ae0 0x7f5e741986e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:38822/0 (socket says 172.21.15.142:38822) 2024-09-06T14:28:55.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:55.397+0000 7f5e73fff640 1 -- 172.21.15.142:0/3293930865 learned_addr learned my addr 172.21.15.142:0/3293930865 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:28:55.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:55.397+0000 7f5e737fe640 1 --2- 172.21.15.142:0/3293930865 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e741084f0 0x7f5e74198c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:28:55.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:55.398+0000 7f5e78cf8640 1 -- 172.21.15.142:0/3293930865 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e741084f0 msgr2=0x7f5e74198c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:28:55.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:55.398+0000 7f5e78cf8640 1 --2- 172.21.15.142:0/3293930865 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e741084f0 0x7f5e74198c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:55.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:55.398+0000 7f5e78cf8640 1 -- 172.21.15.142:0/3293930865 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5e74102ae0 msgr2=0x7f5e741986e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:28:55.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:55.398+0000 7f5e78cf8640 1 --2- 172.21.15.142:0/3293930865 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5e74102ae0 0x7f5e741986e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:55.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:55.398+0000 7f5e78cf8640 1 -- 172.21.15.142:0/3293930865 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e7419e790 con 0x7f5e74108e70 2024-09-06T14:28:55.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:55.398+0000 7f5e73fff640 1 --2- 172.21.15.142:0/3293930865 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5e74102ae0 0x7f5e741986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-06T14:28:55.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:55.398+0000 7f5e78cf8640 1 --2- 172.21.15.142:0/3293930865 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5e74108e70 0x7f5e7419dfc0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f5e6c00ec50 tx=0x7f5e6c00c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:28:55.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:55.398+0000 7f5e717fa640 1 -- 172.21.15.142:0/3293930865 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e6c003b70 con 0x7f5e74108e70 2024-09-06T14:28:55.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:55.398+0000 7f5e717fa640 1 -- 172.21.15.142:0/3293930865 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7f5e6c003d10 con 0x7f5e74108e70 2024-09-06T14:28:55.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:55.398+0000 7f5e7a75b640 1 -- 172.21.15.142:0/3293930865 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e7419ea80 con 0x7f5e74108e70 2024-09-06T14:28:55.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:55.398+0000 7f5e717fa640 1 -- 172.21.15.142:0/3293930865 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e6c010650 con 0x7f5e74108e70 2024-09-06T14:28:55.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:55.398+0000 7f5e7a75b640 1 -- 172.21.15.142:0/3293930865 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e741a5140 con 0x7f5e74108e70 2024-09-06T14:28:55.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:55.399+0000 7f5e7a75b640 1 -- 172.21.15.142:0/3293930865 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e74102ec0 con 0x7f5e74108e70 2024-09-06T14:28:55.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:55.399+0000 7f5e717fa640 1 -- 172.21.15.142:0/3293930865 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 22) ==== 99844+0+0 (secure 0 0 0) 0x7f5e6c004020 con 0x7f5e74108e70 2024-09-06T14:28:55.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:55.400+0000 7f5e717fa640 1 --2- 172.21.15.142:0/3293930865 >> v2:172.21.15.142:6800/2944545612 conn(0x7f5e5c077620 0x7f5e5c079ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:28:55.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:55.400+0000 7f5e717fa640 1 -- 172.21.15.142:0/3293930865 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(64..64 src has 1..64) ==== 6130+0+0 (secure 0 0 0) 0x7f5e6c099ec0 con 0x7f5e74108e70 2024-09-06T14:28:55.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:55.400+0000 7f5e73fff640 1 --2- 172.21.15.142:0/3293930865 >> v2:172.21.15.142:6800/2944545612 conn(0x7f5e5c077620 0x7f5e5c079ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:28:55.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:55.400+0000 7f5e73fff640 1 --2- 172.21.15.142:0/3293930865 >> v2:172.21.15.142:6800/2944545612 conn(0x7f5e5c077620 0x7f5e5c079ae0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f5e640096c0 tx=0x7f5e64005a20 comp rx=0 tx=0).ready entity=mgr.14601 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:28:55.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:55.404+0000 7f5e717fa640 1 -- 172.21.15.142:0/3293930865 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7f5e6c0670f0 con 0x7f5e74108e70 2024-09-06T14:28:55.547 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:28:55 smithi162 ceph-mon[33787]: pgmap v124: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:28:55.589 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:55.592+0000 7f5e7a75b640 1 -- 172.21.15.142:0/3293930865 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f5e7410ac50 con 0x7f5e74108e70 2024-09-06T14:28:55.590 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:55.592+0000 7f5e717fa640 1 -- 172.21.15.142:0/3293930865 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+212 (secure 0 0 0) 0x7f5e6c06b430 con 0x7f5e74108e70 2024-09-06T14:28:55.590 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:28:55.590 INFO:teuthology.orchestra.run.smithi142.stdout:{"status":"HEALTH_WARN","checks":{"MGR_MODULE_DEPENDENCY":{"severity":"HEALTH_WARN","summary":{"message":"Module 'volumes' has failed dependency: No module named 'ceph.fs'","count":1},"muted":false}},"mutes":[]} 2024-09-06T14:28:55.592 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:55.594+0000 7f5e7a75b640 1 -- 172.21.15.142:0/3293930865 >> v2:172.21.15.142:6800/2944545612 conn(0x7f5e5c077620 msgr2=0x7f5e5c079ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:28:55.592 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:55.594+0000 7f5e7a75b640 1 --2- 172.21.15.142:0/3293930865 >> v2:172.21.15.142:6800/2944545612 conn(0x7f5e5c077620 0x7f5e5c079ae0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f5e640096c0 tx=0x7f5e64005a20 comp rx=0 tx=0).stop 2024-09-06T14:28:55.592 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:55.594+0000 7f5e7a75b640 1 -- 172.21.15.142:0/3293930865 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5e74108e70 msgr2=0x7f5e7419dfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:28:55.592 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:55.594+0000 7f5e7a75b640 1 --2- 172.21.15.142:0/3293930865 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5e74108e70 0x7f5e7419dfc0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f5e6c00ec50 tx=0x7f5e6c00c6a0 comp rx=0 tx=0).stop 2024-09-06T14:28:55.592 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:55.594+0000 7f5e7a75b640 1 -- 172.21.15.142:0/3293930865 shutdown_connections 2024-09-06T14:28:55.592 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:55.594+0000 7f5e7a75b640 1 --2- 172.21.15.142:0/3293930865 >> v2:172.21.15.142:6800/2944545612 conn(0x7f5e5c077620 0x7f5e5c079ae0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:55.592 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:55.594+0000 7f5e7a75b640 1 --2- 172.21.15.142:0/3293930865 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5e74108e70 0x7f5e7419dfc0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:55.592 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:55.594+0000 7f5e7a75b640 1 --2- 172.21.15.142:0/3293930865 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e741084f0 0x7f5e74198c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:55.593 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:55.594+0000 7f5e7a75b640 1 --2- 172.21.15.142:0/3293930865 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5e74102ae0 0x7f5e741986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:28:55.593 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:55.594+0000 7f5e7a75b640 1 -- 172.21.15.142:0/3293930865 >> 172.21.15.142:0/3293930865 conn(0x7f5e740fe530 msgr2=0x7f5e74100920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:28:55.593 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:55.594+0000 7f5e7a75b640 1 -- 172.21.15.142:0/3293930865 shutdown_connections 2024-09-06T14:28:55.593 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:28:55.595+0000 7f5e7a75b640 1 -- 172.21.15.142:0/3293930865 wait complete. 2024-09-06T14:28:56.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:28:56 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/3293930865' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:28:56.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:28:56 smithi142 ceph-mon[29958]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-06T14:28:56.210 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:28:56 smithi142 ceph-mon[37372]: from='client.? 172.21.15.142:0/3293930865' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:28:56.210 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:28:56 smithi142 ceph-mon[37372]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-06T14:28:56.548 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:28:56 smithi162 ceph-mon[33787]: from='client.? 172.21.15.142:0/3293930865' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:28:56.548 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:28:56 smithi162 ceph-mon[33787]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-06T14:28:57.459 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:28:57 smithi142 ceph-mon[29958]: pgmap v125: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:28:57.459 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:28:57 smithi142 ceph-mon[37372]: pgmap v125: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:28:57.547 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:28:57 smithi162 ceph-mon[33787]: pgmap v125: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:28:57.548 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 06 14:28:57 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-iscsi-iscsi-a[59853]: debug there is no tcmu-runner data available 2024-09-06T14:28:59.272 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph health --format=json 2024-09-06T14:28:59.459 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:28:59 smithi142 ceph-mon[29958]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-06T14:28:59.459 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:28:59 smithi142 ceph-mon[29958]: pgmap v126: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:28:59.459 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:28:59 smithi142 ceph-mon[37372]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-06T14:28:59.460 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:28:59 smithi142 ceph-mon[37372]: pgmap v126: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:28:59.532 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:28:59.547 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:28:59 smithi162 ceph-mon[33787]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-06T14:28:59.547 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:28:59 smithi162 ceph-mon[33787]: pgmap v126: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:29:00.048 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:00.021+0000 7f173bc32640 1 -- 172.21.15.142:0/3244555941 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1734107d50 msgr2=0x7f173410e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:29:00.049 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:00.021+0000 7f173bc32640 1 --2- 172.21.15.142:0/3244555941 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1734107d50 0x7f173410e550 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f1724009860 tx=0x7f172402a340 comp rx=0 tx=0).stop 2024-09-06T14:29:00.049 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:00.021+0000 7f173bc32640 1 -- 172.21.15.142:0/3244555941 shutdown_connections 2024-09-06T14:29:00.049 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:00.021+0000 7f173bc32640 1 --2- 172.21.15.142:0/3244555941 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1734107d50 0x7f173410e550 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:00.049 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:00.021+0000 7f173bc32640 1 --2- 172.21.15.142:0/3244555941 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f17341073d0 0x7f1734107810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:00.049 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:00.021+0000 7f173bc32640 1 --2- 172.21.15.142:0/3244555941 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f17341019c0 0x7f1734101da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:00.049 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:00.021+0000 7f173bc32640 1 -- 172.21.15.142:0/3244555941 >> 172.21.15.142:0/3244555941 conn(0x7f17340fd430 msgr2=0x7f17340ff850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:29:00.049 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:00.021+0000 7f173bc32640 1 -- 172.21.15.142:0/3244555941 shutdown_connections 2024-09-06T14:29:00.049 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:00.021+0000 7f173bc32640 1 -- 172.21.15.142:0/3244555941 wait complete. 2024-09-06T14:29:00.049 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:00.021+0000 7f173bc32640 1 Processor -- start 2024-09-06T14:29:00.049 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:00.021+0000 7f173bc32640 1 -- start start 2024-09-06T14:29:00.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:00.021+0000 7f173bc32640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f17341019c0 0x7f17340fd930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:29:00.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:00.022+0000 7f173bc32640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f17341073d0 0x7f17340fde70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:29:00.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:00.022+0000 7f173bc32640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1734107d50 0x7f17340fe3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:29:00.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:00.022+0000 7f173bc32640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f1734111120 con 0x7f17341019c0 2024-09-06T14:29:00.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:00.022+0000 7f173bc32640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1734110fa0 con 0x7f17341073d0 2024-09-06T14:29:00.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:00.022+0000 7f173bc32640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7f17341112a0 con 0x7f1734107d50 2024-09-06T14:29:00.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:00.022+0000 7f17399ce640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f17341019c0 0x7f17340fd930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:29:00.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:00.022+0000 7f17399ce640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f17341019c0 0x7f17340fd930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:58426/0 (socket says 172.21.15.142:58426) 2024-09-06T14:29:00.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:00.022+0000 7f17399ce640 1 -- 172.21.15.142:0/2483979158 learned_addr learned my addr 172.21.15.142:0/2483979158 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:29:00.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:00.022+0000 7f173a1cf640 1 --2- 172.21.15.142:0/2483979158 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1734107d50 0x7f17340fe3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:29:00.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:00.022+0000 7f17391cd640 1 --2- 172.21.15.142:0/2483979158 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f17341073d0 0x7f17340fde70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:29:00.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:00.022+0000 7f17399ce640 1 -- 172.21.15.142:0/2483979158 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1734107d50 msgr2=0x7f17340fe3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:29:00.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:00.022+0000 7f17399ce640 1 --2- 172.21.15.142:0/2483979158 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1734107d50 0x7f17340fe3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:00.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:00.022+0000 7f17399ce640 1 -- 172.21.15.142:0/2483979158 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f17341073d0 msgr2=0x7f17340fde70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:29:00.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:00.022+0000 7f17399ce640 1 --2- 172.21.15.142:0/2483979158 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f17341073d0 0x7f17340fde70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:00.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:00.022+0000 7f17399ce640 1 -- 172.21.15.142:0/2483979158 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17341a6a80 con 0x7f17341019c0 2024-09-06T14:29:00.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:00.022+0000 7f173a1cf640 1 --2- 172.21.15.142:0/2483979158 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1734107d50 0x7f17340fe3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-06T14:29:00.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:00.022+0000 7f17399ce640 1 --2- 172.21.15.142:0/2483979158 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f17341019c0 0x7f17340fd930 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f172c00eab0 tx=0x7f172c00ef70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:29:00.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:00.022+0000 7f1732ffd640 1 -- 172.21.15.142:0/2483979158 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f172c00ce40 con 0x7f17341019c0 2024-09-06T14:29:00.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:00.022+0000 7f1732ffd640 1 -- 172.21.15.142:0/2483979158 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7f172c0118f0 con 0x7f17341019c0 2024-09-06T14:29:00.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:00.022+0000 7f173bc32640 1 -- 172.21.15.142:0/2483979158 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17341a6d70 con 0x7f17341019c0 2024-09-06T14:29:00.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:00.022+0000 7f1732ffd640 1 -- 172.21.15.142:0/2483979158 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f172c0108f0 con 0x7f17341019c0 2024-09-06T14:29:00.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:00.022+0000 7f173bc32640 1 -- 172.21.15.142:0/2483979158 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17341a7250 con 0x7f17341019c0 2024-09-06T14:29:00.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:00.023+0000 7f173bc32640 1 -- 172.21.15.142:0/2483979158 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1734101da0 con 0x7f17341019c0 2024-09-06T14:29:00.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:00.024+0000 7f1732ffd640 1 -- 172.21.15.142:0/2483979158 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 22) ==== 99844+0+0 (secure 0 0 0) 0x7f172c011aa0 con 0x7f17341019c0 2024-09-06T14:29:00.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:00.024+0000 7f1732ffd640 1 --2- 172.21.15.142:0/2483979158 >> v2:172.21.15.142:6800/2944545612 conn(0x7f171c077620 0x7f171c079ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:29:00.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:00.024+0000 7f1732ffd640 1 -- 172.21.15.142:0/2483979158 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 6130+0+0 (secure 0 0 0) 0x7f172c099eb0 con 0x7f17341019c0 2024-09-06T14:29:00.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:00.024+0000 7f17391cd640 1 --2- 172.21.15.142:0/2483979158 >> v2:172.21.15.142:6800/2944545612 conn(0x7f171c077620 0x7f171c079ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:29:00.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:00.025+0000 7f17391cd640 1 --2- 172.21.15.142:0/2483979158 >> v2:172.21.15.142:6800/2944545612 conn(0x7f171c077620 0x7f171c079ae0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f17340ff420 tx=0x7f1728009260 comp rx=0 tx=0).ready entity=mgr.14601 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:29:00.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:00.028+0000 7f1732ffd640 1 -- 172.21.15.142:0/2483979158 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7f172c016030 con 0x7f17341019c0 2024-09-06T14:29:00.217 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:00.220+0000 7f173bc32640 1 -- 172.21.15.142:0/2483979158 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f173410aba0 con 0x7f17341019c0 2024-09-06T14:29:00.217 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:00.220+0000 7f1732ffd640 1 -- 172.21.15.142:0/2483979158 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+212 (secure 0 0 0) 0x7f172c06b420 con 0x7f17341019c0 2024-09-06T14:29:00.218 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:29:00.218 INFO:teuthology.orchestra.run.smithi142.stdout:{"status":"HEALTH_WARN","checks":{"MGR_MODULE_DEPENDENCY":{"severity":"HEALTH_WARN","summary":{"message":"Module 'volumes' has failed dependency: No module named 'ceph.fs'","count":1},"muted":false}},"mutes":[]} 2024-09-06T14:29:00.220 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:00.222+0000 7f173bc32640 1 -- 172.21.15.142:0/2483979158 >> v2:172.21.15.142:6800/2944545612 conn(0x7f171c077620 msgr2=0x7f171c079ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:29:00.220 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:00.222+0000 7f173bc32640 1 --2- 172.21.15.142:0/2483979158 >> v2:172.21.15.142:6800/2944545612 conn(0x7f171c077620 0x7f171c079ae0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f17340ff420 tx=0x7f1728009260 comp rx=0 tx=0).stop 2024-09-06T14:29:00.220 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:00.222+0000 7f173bc32640 1 -- 172.21.15.142:0/2483979158 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f17341019c0 msgr2=0x7f17340fd930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:29:00.220 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:00.222+0000 7f173bc32640 1 --2- 172.21.15.142:0/2483979158 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f17341019c0 0x7f17340fd930 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f172c00eab0 tx=0x7f172c00ef70 comp rx=0 tx=0).stop 2024-09-06T14:29:00.220 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:00.222+0000 7f173bc32640 1 -- 172.21.15.142:0/2483979158 shutdown_connections 2024-09-06T14:29:00.220 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:00.222+0000 7f173bc32640 1 --2- 172.21.15.142:0/2483979158 >> v2:172.21.15.142:6800/2944545612 conn(0x7f171c077620 0x7f171c079ae0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:00.220 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:00.222+0000 7f173bc32640 1 --2- 172.21.15.142:0/2483979158 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1734107d50 0x7f17340fe3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:00.220 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:00.222+0000 7f173bc32640 1 --2- 172.21.15.142:0/2483979158 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f17341073d0 0x7f17340fde70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:00.221 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:00.222+0000 7f173bc32640 1 --2- 172.21.15.142:0/2483979158 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f17341019c0 0x7f17340fd930 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:00.221 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:00.222+0000 7f173bc32640 1 -- 172.21.15.142:0/2483979158 >> 172.21.15.142:0/2483979158 conn(0x7f17340fd430 msgr2=0x7f173406a9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:29:00.221 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:00.222+0000 7f173bc32640 1 -- 172.21.15.142:0/2483979158 shutdown_connections 2024-09-06T14:29:00.221 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:00.222+0000 7f173bc32640 1 -- 172.21.15.142:0/2483979158 wait complete. 2024-09-06T14:29:01.459 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:29:01 smithi142 ceph-mon[29958]: pgmap v127: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:29:01.459 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:29:01 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/2483979158' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:29:01.460 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:29:01 smithi142 ceph-mon[37372]: pgmap v127: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:29:01.460 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:29:01 smithi142 ceph-mon[37372]: from='client.? 172.21.15.142:0/2483979158' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:29:01.548 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:29:01 smithi162 ceph-mon[33787]: pgmap v127: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:29:01.548 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:29:01 smithi162 ceph-mon[33787]: from='client.? 172.21.15.142:0/2483979158' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:29:03.459 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:29:03 smithi142 ceph-mon[29958]: pgmap v128: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:29:03.459 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:29:03 smithi142 ceph-mon[37372]: pgmap v128: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:29:03.547 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:29:03 smithi162 ceph-mon[33787]: pgmap v128: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:29:03.813 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph health --format=json 2024-09-06T14:29:04.074 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:29:04.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:04.542+0000 7f6491834640 1 -- 172.21.15.142:0/1504503626 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f648c108e60 msgr2=0x7f648c10f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:29:04.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:04.542+0000 7f6491834640 1 --2- 172.21.15.142:0/1504503626 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f648c108e60 0x7f648c10f660 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f6484009900 tx=0x7f648402a410 comp rx=0 tx=0).stop 2024-09-06T14:29:04.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:04.542+0000 7f6491834640 1 -- 172.21.15.142:0/1504503626 shutdown_connections 2024-09-06T14:29:04.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:04.542+0000 7f6491834640 1 --2- 172.21.15.142:0/1504503626 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f648c108e60 0x7f648c10f660 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:04.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:04.542+0000 7f6491834640 1 --2- 172.21.15.142:0/1504503626 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f648c1084e0 0x7f648c108920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:04.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:04.542+0000 7f6491834640 1 --2- 172.21.15.142:0/1504503626 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f648c102ad0 0x7f648c102eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:04.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:04.542+0000 7f6491834640 1 -- 172.21.15.142:0/1504503626 >> 172.21.15.142:0/1504503626 conn(0x7f648c0fe580 msgr2=0x7f648c1009a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:29:04.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:04.542+0000 7f6491834640 1 -- 172.21.15.142:0/1504503626 shutdown_connections 2024-09-06T14:29:04.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:04.542+0000 7f6491834640 1 -- 172.21.15.142:0/1504503626 wait complete. 2024-09-06T14:29:04.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:04.543+0000 7f6491834640 1 Processor -- start 2024-09-06T14:29:04.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:04.543+0000 7f6491834640 1 -- start start 2024-09-06T14:29:04.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:04.543+0000 7f6491834640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f648c102ad0 0x7f648c198640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:29:04.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:04.543+0000 7f6491834640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f648c1084e0 0x7f648c198b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:29:04.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:04.543+0000 7f6491834640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f648c108e60 0x7f648c19df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:29:04.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:04.543+0000 7f6491834640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f648c1121f0 con 0x7f648c102ad0 2024-09-06T14:29:04.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:04.543+0000 7f6491834640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f648c112070 con 0x7f648c108e60 2024-09-06T14:29:04.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:04.543+0000 7f6491834640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7f648c112370 con 0x7f648c1084e0 2024-09-06T14:29:04.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:04.543+0000 7f648affd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f648c102ad0 0x7f648c198640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:29:04.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:04.543+0000 7f648a7fc640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f648c1084e0 0x7f648c198b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:29:04.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:04.543+0000 7f648affd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f648c102ad0 0x7f648c198640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:58444/0 (socket says 172.21.15.142:58444) 2024-09-06T14:29:04.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:04.543+0000 7f648affd640 1 -- 172.21.15.142:0/89915422 learned_addr learned my addr 172.21.15.142:0/89915422 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:29:04.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:04.544+0000 7f648b7fe640 1 --2- 172.21.15.142:0/89915422 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f648c108e60 0x7f648c19df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:29:04.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:04.544+0000 7f648affd640 1 -- 172.21.15.142:0/89915422 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f648c1084e0 msgr2=0x7f648c198b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:29:04.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:04.544+0000 7f648affd640 1 --2- 172.21.15.142:0/89915422 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f648c1084e0 0x7f648c198b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:04.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:04.544+0000 7f648affd640 1 -- 172.21.15.142:0/89915422 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f648c108e60 msgr2=0x7f648c19df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:29:04.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:04.544+0000 7f648affd640 1 --2- 172.21.15.142:0/89915422 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f648c108e60 0x7f648c19df20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:04.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:04.544+0000 7f648affd640 1 -- 172.21.15.142:0/89915422 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f648c19e6f0 con 0x7f648c102ad0 2024-09-06T14:29:04.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:04.544+0000 7f648affd640 1 --2- 172.21.15.142:0/89915422 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f648c102ad0 0x7f648c198640 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f647c009d50 tx=0x7f647c00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:29:04.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:04.544+0000 7f6490832640 1 -- 172.21.15.142:0/89915422 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f647c003b70 con 0x7f648c102ad0 2024-09-06T14:29:04.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:04.544+0000 7f6490832640 1 -- 172.21.15.142:0/89915422 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7f647c003d10 con 0x7f648c102ad0 2024-09-06T14:29:04.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:04.544+0000 7f6491834640 1 -- 172.21.15.142:0/89915422 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f648c19e9e0 con 0x7f648c102ad0 2024-09-06T14:29:04.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:04.544+0000 7f6490832640 1 -- 172.21.15.142:0/89915422 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f647c0107d0 con 0x7f648c102ad0 2024-09-06T14:29:04.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:04.544+0000 7f6491834640 1 -- 172.21.15.142:0/89915422 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f648c1a50a0 con 0x7f648c102ad0 2024-09-06T14:29:04.548 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:04.545+0000 7f6491834640 1 -- 172.21.15.142:0/89915422 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f648c102eb0 con 0x7f648c102ad0 2024-09-06T14:29:04.548 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:04.546+0000 7f6490832640 1 -- 172.21.15.142:0/89915422 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 22) ==== 99844+0+0 (secure 0 0 0) 0x7f647c004020 con 0x7f648c102ad0 2024-09-06T14:29:04.548 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:04.546+0000 7f6490832640 1 --2- 172.21.15.142:0/89915422 >> v2:172.21.15.142:6800/2944545612 conn(0x7f6474077620 0x7f6474079ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:29:04.548 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:04.546+0000 7f648a7fc640 1 --2- 172.21.15.142:0/89915422 >> v2:172.21.15.142:6800/2944545612 conn(0x7f6474077620 0x7f6474079ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:29:04.548 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:04.546+0000 7f6490832640 1 -- 172.21.15.142:0/89915422 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 6130+0+0 (secure 0 0 0) 0x7f647c09a0c0 con 0x7f648c102ad0 2024-09-06T14:29:04.548 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:04.547+0000 7f648a7fc640 1 --2- 172.21.15.142:0/89915422 >> v2:172.21.15.142:6800/2944545612 conn(0x7f6474077620 0x7f6474079ae0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f648c107f30 tx=0x7f648000b390 comp rx=0 tx=0).ready entity=mgr.14601 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:29:04.548 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:04.550+0000 7f6490832640 1 -- 172.21.15.142:0/89915422 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7f647c063180 con 0x7f648c102ad0 2024-09-06T14:29:04.709 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:29:04 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: ::ffff:172.21.15.162 - - [06/Sep/2024:14:29:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-06T14:29:04.736 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:04.738+0000 7f6491834640 1 -- 172.21.15.142:0/89915422 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f648c10ac40 con 0x7f648c102ad0 2024-09-06T14:29:04.736 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:04.739+0000 7f6490832640 1 -- 172.21.15.142:0/89915422 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+212 (secure 0 0 0) 0x7f647c0672f0 con 0x7f648c102ad0 2024-09-06T14:29:04.736 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:29:04.737 INFO:teuthology.orchestra.run.smithi142.stdout:{"status":"HEALTH_WARN","checks":{"MGR_MODULE_DEPENDENCY":{"severity":"HEALTH_WARN","summary":{"message":"Module 'volumes' has failed dependency: No module named 'ceph.fs'","count":1},"muted":false}},"mutes":[]} 2024-09-06T14:29:04.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:04.741+0000 7f6491834640 1 -- 172.21.15.142:0/89915422 >> v2:172.21.15.142:6800/2944545612 conn(0x7f6474077620 msgr2=0x7f6474079ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:29:04.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:04.741+0000 7f6491834640 1 --2- 172.21.15.142:0/89915422 >> v2:172.21.15.142:6800/2944545612 conn(0x7f6474077620 0x7f6474079ae0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f648c107f30 tx=0x7f648000b390 comp rx=0 tx=0).stop 2024-09-06T14:29:04.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:04.741+0000 7f6491834640 1 -- 172.21.15.142:0/89915422 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f648c102ad0 msgr2=0x7f648c198640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:29:04.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:04.741+0000 7f6491834640 1 --2- 172.21.15.142:0/89915422 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f648c102ad0 0x7f648c198640 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f647c009d50 tx=0x7f647c00c6a0 comp rx=0 tx=0).stop 2024-09-06T14:29:04.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:04.741+0000 7f6491834640 1 -- 172.21.15.142:0/89915422 shutdown_connections 2024-09-06T14:29:04.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:04.741+0000 7f6491834640 1 --2- 172.21.15.142:0/89915422 >> v2:172.21.15.142:6800/2944545612 conn(0x7f6474077620 0x7f6474079ae0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:04.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:04.741+0000 7f6491834640 1 --2- 172.21.15.142:0/89915422 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f648c108e60 0x7f648c19df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:04.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:04.741+0000 7f6491834640 1 --2- 172.21.15.142:0/89915422 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f648c1084e0 0x7f648c198b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:04.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:04.741+0000 7f6491834640 1 --2- 172.21.15.142:0/89915422 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f648c102ad0 0x7f648c198640 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:04.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:04.741+0000 7f6491834640 1 -- 172.21.15.142:0/89915422 >> 172.21.15.142:0/89915422 conn(0x7f648c0fe580 msgr2=0x7f648c100970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:29:04.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:04.741+0000 7f6491834640 1 -- 172.21.15.142:0/89915422 shutdown_connections 2024-09-06T14:29:04.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:04.741+0000 7f6491834640 1 -- 172.21.15.142:0/89915422 wait complete. 2024-09-06T14:29:05.376 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:29:05 smithi142 ceph-mon[29958]: pgmap v129: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:29:05.376 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:29:05 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/89915422' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:29:05.376 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:29:05 smithi142 ceph-mon[37372]: pgmap v129: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:29:05.376 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:29:05 smithi142 ceph-mon[37372]: from='client.? 172.21.15.142:0/89915422' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:29:05.547 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:29:05 smithi162 ceph-mon[33787]: pgmap v129: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:29:05.547 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:29:05 smithi162 ceph-mon[33787]: from='client.? 172.21.15.142:0/89915422' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:29:06.459 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:29:06 smithi142 ceph-mon[29958]: pgmap v130: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:29:06.459 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:29:06 smithi142 ceph-mon[37372]: pgmap v130: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:29:06.547 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:29:06 smithi162 ceph-mon[33787]: pgmap v130: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:29:07.547 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 06 14:29:07 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-iscsi-iscsi-a[59853]: debug there is no tcmu-runner data available 2024-09-06T14:29:08.378 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph health --format=json 2024-09-06T14:29:08.639 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:29:08.709 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:29:08 smithi142 ceph-mon[29958]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-06T14:29:08.709 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:29:08 smithi142 ceph-mon[29958]: pgmap v131: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:29:08.710 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:29:08 smithi142 ceph-mon[37372]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-06T14:29:08.710 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:29:08 smithi142 ceph-mon[37372]: pgmap v131: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:29:09.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:29:08 smithi162 ceph-mon[33787]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-06T14:29:09.048 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:29:08 smithi162 ceph-mon[33787]: pgmap v131: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:29:09.101 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:09.097+0000 7f1b9d1f6640 1 -- 172.21.15.142:0/1622696134 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1b9810ce30 msgr2=0x7f1b9810f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:29:09.101 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:09.097+0000 7f1b9d1f6640 1 --2- 172.21.15.142:0/1622696134 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1b9810ce30 0x7f1b9810f290 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f1b90009ca0 tx=0x7f1b9002a8c0 comp rx=0 tx=0).stop 2024-09-06T14:29:09.101 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:09.097+0000 7f1b9d1f6640 1 -- 172.21.15.142:0/1622696134 shutdown_connections 2024-09-06T14:29:09.101 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:09.097+0000 7f1b9d1f6640 1 --2- 172.21.15.142:0/1622696134 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1b9810ce30 0x7f1b9810f290 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:09.101 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:09.097+0000 7f1b9d1f6640 1 --2- 172.21.15.142:0/1622696134 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1b981042f0 0x7f1b9810c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:09.101 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:09.097+0000 7f1b9d1f6640 1 --2- 172.21.15.142:0/1622696134 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1b981039d0 0x7f1b98103db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:09.101 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:09.097+0000 7f1b9d1f6640 1 -- 172.21.15.142:0/1622696134 >> 172.21.15.142:0/1622696134 conn(0x7f1b980fd430 msgr2=0x7f1b980ff850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:29:09.101 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:09.097+0000 7f1b9d1f6640 1 -- 172.21.15.142:0/1622696134 shutdown_connections 2024-09-06T14:29:09.101 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:09.097+0000 7f1b9d1f6640 1 -- 172.21.15.142:0/1622696134 wait complete. 2024-09-06T14:29:09.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:09.098+0000 7f1b9d1f6640 1 Processor -- start 2024-09-06T14:29:09.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:09.098+0000 7f1b9d1f6640 1 -- start start 2024-09-06T14:29:09.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:09.098+0000 7f1b9d1f6640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1b981039d0 0x7f1b98198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:29:09.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:09.098+0000 7f1b9d1f6640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1b981042f0 0x7f1b98198c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:29:09.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:09.098+0000 7f1b9d1f6640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1b9810ce30 0x7f1b9819e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:29:09.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:09.098+0000 7f1b9d1f6640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f1b981121b0 con 0x7f1b981042f0 2024-09-06T14:29:09.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:09.098+0000 7f1b9d1f6640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1b98112030 con 0x7f1b981039d0 2024-09-06T14:29:09.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:09.098+0000 7f1b9d1f6640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7f1b98112330 con 0x7f1b9810ce30 2024-09-06T14:29:09.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:09.098+0000 7f1b9759e640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1b9810ce30 0x7f1b9819e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:29:09.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:09.098+0000 7f1b9659c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1b981042f0 0x7f1b98198c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:29:09.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:09.098+0000 7f1b9759e640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1b9810ce30 0x7f1b9819e010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:45392/0 (socket says 172.21.15.142:45392) 2024-09-06T14:29:09.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:09.098+0000 7f1b9759e640 1 -- 172.21.15.142:0/1764426216 learned_addr learned my addr 172.21.15.142:0/1764426216 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:29:09.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:09.098+0000 7f1b96d9d640 1 --2- 172.21.15.142:0/1764426216 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1b981039d0 0x7f1b98198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:29:09.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:09.098+0000 7f1b9759e640 1 -- 172.21.15.142:0/1764426216 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1b981039d0 msgr2=0x7f1b98198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:29:09.103 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:09.098+0000 7f1b9759e640 1 --2- 172.21.15.142:0/1764426216 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1b981039d0 0x7f1b98198730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:09.103 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:09.098+0000 7f1b9759e640 1 -- 172.21.15.142:0/1764426216 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1b981042f0 msgr2=0x7f1b98198c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:29:09.103 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:09.098+0000 7f1b9759e640 1 --2- 172.21.15.142:0/1764426216 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1b981042f0 0x7f1b98198c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:09.103 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:09.098+0000 7f1b9759e640 1 -- 172.21.15.142:0/1764426216 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b9819e7e0 con 0x7f1b9810ce30 2024-09-06T14:29:09.103 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:09.099+0000 7f1b9759e640 1 --2- 172.21.15.142:0/1764426216 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1b9810ce30 0x7f1b9819e010 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f1b90005cc0 tx=0x7f1b900093e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:29:09.103 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:09.099+0000 7f1b7ffff640 1 -- 172.21.15.142:0/1764426216 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b90003cf0 con 0x7f1b9810ce30 2024-09-06T14:29:09.103 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:09.099+0000 7f1b9d1f6640 1 -- 172.21.15.142:0/1764426216 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b9819ea70 con 0x7f1b9810ce30 2024-09-06T14:29:09.103 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:09.099+0000 7f1b7ffff640 1 -- 172.21.15.142:0/1764426216 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7f1b900044c0 con 0x7f1b9810ce30 2024-09-06T14:29:09.103 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:09.099+0000 7f1b7ffff640 1 -- 172.21.15.142:0/1764426216 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b90030b20 con 0x7f1b9810ce30 2024-09-06T14:29:09.103 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:09.099+0000 7f1b9d1f6640 1 -- 172.21.15.142:0/1764426216 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b981a5190 con 0x7f1b9810ce30 2024-09-06T14:29:09.103 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:09.100+0000 7f1b9d1f6640 1 -- 172.21.15.142:0/1764426216 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b98103db0 con 0x7f1b9810ce30 2024-09-06T14:29:09.103 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:09.101+0000 7f1b7ffff640 1 -- 172.21.15.142:0/1764426216 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 22) ==== 99844+0+0 (secure 0 0 0) 0x7f1b90030cc0 con 0x7f1b9810ce30 2024-09-06T14:29:09.103 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:09.101+0000 7f1b7ffff640 1 --2- 172.21.15.142:0/1764426216 >> v2:172.21.15.142:6800/2944545612 conn(0x7f1b80077620 0x7f1b80079ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:29:09.103 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:09.101+0000 7f1b7ffff640 1 -- 172.21.15.142:0/1764426216 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(64..64 src has 1..64) ==== 6130+0+0 (secure 0 0 0) 0x7f1b900bace0 con 0x7f1b9810ce30 2024-09-06T14:29:09.104 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:09.101+0000 7f1b96d9d640 1 --2- 172.21.15.142:0/1764426216 >> v2:172.21.15.142:6800/2944545612 conn(0x7f1b80077620 0x7f1b80079ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:29:09.104 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:09.102+0000 7f1b96d9d640 1 --2- 172.21.15.142:0/1764426216 >> v2:172.21.15.142:6800/2944545612 conn(0x7f1b80077620 0x7f1b80079ae0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f1b880058c0 tx=0x7f1b8800a250 comp rx=0 tx=0).ready entity=mgr.14601 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:29:09.104 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:09.105+0000 7f1b7ffff640 1 -- 172.21.15.142:0/1764426216 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7f1b90088020 con 0x7f1b9810ce30 2024-09-06T14:29:09.295 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:09.296+0000 7f1b9d1f6640 1 -- 172.21.15.142:0/1764426216 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f1b98107500 con 0x7f1b9810ce30 2024-09-06T14:29:09.295 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:09.297+0000 7f1b7ffff640 1 -- 172.21.15.142:0/1764426216 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+212 (secure 0 0 0) 0x7f1b9008ca40 con 0x7f1b9810ce30 2024-09-06T14:29:09.295 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:29:09.296 INFO:teuthology.orchestra.run.smithi142.stdout:{"status":"HEALTH_WARN","checks":{"MGR_MODULE_DEPENDENCY":{"severity":"HEALTH_WARN","summary":{"message":"Module 'volumes' has failed dependency: No module named 'ceph.fs'","count":1},"muted":false}},"mutes":[]} 2024-09-06T14:29:09.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:09.299+0000 7f1b9d1f6640 1 -- 172.21.15.142:0/1764426216 >> v2:172.21.15.142:6800/2944545612 conn(0x7f1b80077620 msgr2=0x7f1b80079ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:29:09.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:09.299+0000 7f1b9d1f6640 1 --2- 172.21.15.142:0/1764426216 >> v2:172.21.15.142:6800/2944545612 conn(0x7f1b80077620 0x7f1b80079ae0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f1b880058c0 tx=0x7f1b8800a250 comp rx=0 tx=0).stop 2024-09-06T14:29:09.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:09.299+0000 7f1b9d1f6640 1 -- 172.21.15.142:0/1764426216 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1b9810ce30 msgr2=0x7f1b9819e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:29:09.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:09.299+0000 7f1b9d1f6640 1 --2- 172.21.15.142:0/1764426216 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1b9810ce30 0x7f1b9819e010 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f1b90005cc0 tx=0x7f1b900093e0 comp rx=0 tx=0).stop 2024-09-06T14:29:09.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:09.299+0000 7f1b9d1f6640 1 -- 172.21.15.142:0/1764426216 shutdown_connections 2024-09-06T14:29:09.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:09.299+0000 7f1b9d1f6640 1 --2- 172.21.15.142:0/1764426216 >> v2:172.21.15.142:6800/2944545612 conn(0x7f1b80077620 0x7f1b80079ae0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:09.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:09.299+0000 7f1b9d1f6640 1 --2- 172.21.15.142:0/1764426216 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1b9810ce30 0x7f1b9819e010 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:09.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:09.299+0000 7f1b9d1f6640 1 --2- 172.21.15.142:0/1764426216 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1b981042f0 0x7f1b98198c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:09.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:09.299+0000 7f1b9d1f6640 1 --2- 172.21.15.142:0/1764426216 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1b981039d0 0x7f1b98198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:09.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:09.299+0000 7f1b9d1f6640 1 -- 172.21.15.142:0/1764426216 >> 172.21.15.142:0/1764426216 conn(0x7f1b980fd430 msgr2=0x7f1b981046d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:29:09.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:09.299+0000 7f1b9d1f6640 1 -- 172.21.15.142:0/1764426216 shutdown_connections 2024-09-06T14:29:09.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:09.299+0000 7f1b9d1f6640 1 -- 172.21.15.142:0/1764426216 wait complete. 2024-09-06T14:29:09.905 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:29:09 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/1764426216' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:29:09.905 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:29:09 smithi142 ceph-mon[37372]: from='client.? 172.21.15.142:0/1764426216' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:29:10.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:29:09 smithi162 ceph-mon[33787]: from='client.? 172.21.15.142:0/1764426216' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:29:10.960 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:29:10 smithi142 ceph-mon[29958]: pgmap v132: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:29:10.960 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:29:10 smithi142 ceph-mon[37372]: pgmap v132: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:29:11.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:29:10 smithi162 ceph-mon[33787]: pgmap v132: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:29:11.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:29:11 smithi142 ceph-mon[29958]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-06T14:29:12.057 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:29:11 smithi142 ceph-mon[37372]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-06T14:29:12.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:29:11 smithi162 ceph-mon[33787]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-06T14:29:12.939 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph health --format=json 2024-09-06T14:29:12.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:29:12 smithi142 ceph-mon[29958]: pgmap v133: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:29:12.959 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:29:12 smithi142 ceph-mon[37372]: pgmap v133: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:29:13.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:29:12 smithi162 ceph-mon[33787]: pgmap v133: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:29:13.199 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:29:13.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:13.647+0000 7f9d8a7a0640 1 -- 172.21.15.142:0/1589398518 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9d8410cc50 msgr2=0x7f9d8410f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:29:13.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:13.647+0000 7f9d8a7a0640 1 --2- 172.21.15.142:0/1589398518 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9d8410cc50 0x7f9d8410f0d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f9d7c009be0 tx=0x7f9d7c02a6c0 comp rx=0 tx=0).stop 2024-09-06T14:29:13.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:13.647+0000 7f9d8a7a0640 1 -- 172.21.15.142:0/1589398518 shutdown_connections 2024-09-06T14:29:13.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:13.647+0000 7f9d8a7a0640 1 --2- 172.21.15.142:0/1589398518 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9d8410cc50 0x7f9d8410f0d0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:13.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:13.647+0000 7f9d8a7a0640 1 --2- 172.21.15.142:0/1589398518 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9d841040a0 0x7f9d8410c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:13.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:13.647+0000 7f9d8a7a0640 1 --2- 172.21.15.142:0/1589398518 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9d841036a0 0x7f9d84103a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:13.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:13.647+0000 7f9d8a7a0640 1 -- 172.21.15.142:0/1589398518 >> 172.21.15.142:0/1589398518 conn(0x7f9d840fd2d0 msgr2=0x7f9d840ff6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:29:13.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:13.647+0000 7f9d8a7a0640 1 -- 172.21.15.142:0/1589398518 shutdown_connections 2024-09-06T14:29:13.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:13.647+0000 7f9d8a7a0640 1 -- 172.21.15.142:0/1589398518 wait complete. 2024-09-06T14:29:13.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:13.648+0000 7f9d8a7a0640 1 Processor -- start 2024-09-06T14:29:13.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:13.648+0000 7f9d8a7a0640 1 -- start start 2024-09-06T14:29:13.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:13.648+0000 7f9d8a7a0640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9d841036a0 0x7f9d84198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:29:13.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:13.648+0000 7f9d8a7a0640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9d841040a0 0x7f9d84198a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:29:13.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:13.648+0000 7f9d8a7a0640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9d8410cc50 0x7f9d8419de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:29:13.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:13.648+0000 7f9d8a7a0640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f9d84111ff0 con 0x7f9d841040a0 2024-09-06T14:29:13.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:13.648+0000 7f9d8a7a0640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f9d84111e70 con 0x7f9d841036a0 2024-09-06T14:29:13.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:13.648+0000 7f9d8a7a0640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7f9d84112170 con 0x7f9d8410cc50 2024-09-06T14:29:13.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:13.648+0000 7f9d88f9d640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9d841040a0 0x7f9d84198a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:29:13.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:13.648+0000 7f9d89f9f640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9d8410cc50 0x7f9d8419de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:29:13.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:13.648+0000 7f9d89f9f640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9d8410cc50 0x7f9d8419de10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:45414/0 (socket says 172.21.15.142:45414) 2024-09-06T14:29:13.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:13.648+0000 7f9d88f9d640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9d841040a0 0x7f9d84198a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:37642/0 (socket says 172.21.15.142:37642) 2024-09-06T14:29:13.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:13.648+0000 7f9d89f9f640 1 -- 172.21.15.142:0/3606920009 learned_addr learned my addr 172.21.15.142:0/3606920009 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:29:13.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:13.648+0000 7f9d8979e640 1 --2- 172.21.15.142:0/3606920009 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9d841036a0 0x7f9d84198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:29:13.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:13.649+0000 7f9d89f9f640 1 -- 172.21.15.142:0/3606920009 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9d841036a0 msgr2=0x7f9d84198530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:29:13.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:13.649+0000 7f9d89f9f640 1 --2- 172.21.15.142:0/3606920009 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9d841036a0 0x7f9d84198530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:13.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:13.649+0000 7f9d89f9f640 1 -- 172.21.15.142:0/3606920009 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9d841040a0 msgr2=0x7f9d84198a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:29:13.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:13.649+0000 7f9d89f9f640 1 --2- 172.21.15.142:0/3606920009 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9d841040a0 0x7f9d84198a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:13.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:13.649+0000 7f9d89f9f640 1 -- 172.21.15.142:0/3606920009 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d8419e5e0 con 0x7f9d8410cc50 2024-09-06T14:29:13.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:13.649+0000 7f9d88f9d640 1 --2- 172.21.15.142:0/3606920009 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9d841040a0 0x7f9d84198a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-06T14:29:13.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:13.649+0000 7f9d89f9f640 1 --2- 172.21.15.142:0/3606920009 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9d8410cc50 0x7f9d8419de10 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f9d7c004af0 tx=0x7f9d7c030650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:29:13.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:13.649+0000 7f9d7a7fc640 1 -- 172.21.15.142:0/3606920009 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d7c041020 con 0x7f9d8410cc50 2024-09-06T14:29:13.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:13.649+0000 7f9d8a7a0640 1 -- 172.21.15.142:0/3606920009 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d8419e870 con 0x7f9d8410cc50 2024-09-06T14:29:13.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:13.649+0000 7f9d7a7fc640 1 -- 172.21.15.142:0/3606920009 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7f9d7c003970 con 0x7f9d8410cc50 2024-09-06T14:29:13.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:13.649+0000 7f9d7a7fc640 1 -- 172.21.15.142:0/3606920009 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d7c036040 con 0x7f9d8410cc50 2024-09-06T14:29:13.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:13.649+0000 7f9d8a7a0640 1 -- 172.21.15.142:0/3606920009 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d841a4f90 con 0x7f9d8410cc50 2024-09-06T14:29:13.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:13.651+0000 7f9d8a7a0640 1 -- 172.21.15.142:0/3606920009 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d84103a80 con 0x7f9d8410cc50 2024-09-06T14:29:13.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:13.653+0000 7f9d7a7fc640 1 -- 172.21.15.142:0/3606920009 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 22) ==== 99844+0+0 (secure 0 0 0) 0x7f9d7c044020 con 0x7f9d8410cc50 2024-09-06T14:29:13.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:13.653+0000 7f9d7a7fc640 1 --2- 172.21.15.142:0/3606920009 >> v2:172.21.15.142:6800/2944545612 conn(0x7f9d700776d0 0x7f9d70079b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:29:13.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:13.653+0000 7f9d7a7fc640 1 -- 172.21.15.142:0/3606920009 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(64..64 src has 1..64) ==== 6130+0+0 (secure 0 0 0) 0x7f9d7c0baa60 con 0x7f9d8410cc50 2024-09-06T14:29:13.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:13.653+0000 7f9d8979e640 1 --2- 172.21.15.142:0/3606920009 >> v2:172.21.15.142:6800/2944545612 conn(0x7f9d700776d0 0x7f9d70079b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:29:13.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:13.654+0000 7f9d8979e640 1 --2- 172.21.15.142:0/3606920009 >> v2:172.21.15.142:6800/2944545612 conn(0x7f9d700776d0 0x7f9d70079b90 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f9d74009530 tx=0x7f9d74009260 comp rx=0 tx=0).ready entity=mgr.14601 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:29:13.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:13.656+0000 7f9d7a7fc640 1 -- 172.21.15.142:0/3606920009 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7f9d7c04b020 con 0x7f9d8410cc50 2024-09-06T14:29:13.841 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:13.843+0000 7f9d8a7a0640 1 -- 172.21.15.142:0/3606920009 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f9d84199660 con 0x7f9d8410cc50 2024-09-06T14:29:13.841 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:13.844+0000 7f9d7a7fc640 1 -- 172.21.15.142:0/3606920009 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+212 (secure 0 0 0) 0x7f9d84199660 con 0x7f9d8410cc50 2024-09-06T14:29:13.842 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:29:13.842 INFO:teuthology.orchestra.run.smithi142.stdout:{"status":"HEALTH_WARN","checks":{"MGR_MODULE_DEPENDENCY":{"severity":"HEALTH_WARN","summary":{"message":"Module 'volumes' has failed dependency: No module named 'ceph.fs'","count":1},"muted":false}},"mutes":[]} 2024-09-06T14:29:13.843 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:13.846+0000 7f9d8a7a0640 1 -- 172.21.15.142:0/3606920009 >> v2:172.21.15.142:6800/2944545612 conn(0x7f9d700776d0 msgr2=0x7f9d70079b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:29:13.843 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:13.846+0000 7f9d8a7a0640 1 --2- 172.21.15.142:0/3606920009 >> v2:172.21.15.142:6800/2944545612 conn(0x7f9d700776d0 0x7f9d70079b90 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f9d74009530 tx=0x7f9d74009260 comp rx=0 tx=0).stop 2024-09-06T14:29:13.843 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:13.846+0000 7f9d8a7a0640 1 -- 172.21.15.142:0/3606920009 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9d8410cc50 msgr2=0x7f9d8419de10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:29:13.844 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:13.846+0000 7f9d8a7a0640 1 --2- 172.21.15.142:0/3606920009 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9d8410cc50 0x7f9d8419de10 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f9d7c004af0 tx=0x7f9d7c030650 comp rx=0 tx=0).stop 2024-09-06T14:29:13.844 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:13.846+0000 7f9d8a7a0640 1 -- 172.21.15.142:0/3606920009 shutdown_connections 2024-09-06T14:29:13.844 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:13.846+0000 7f9d8a7a0640 1 --2- 172.21.15.142:0/3606920009 >> v2:172.21.15.142:6800/2944545612 conn(0x7f9d700776d0 0x7f9d70079b90 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:13.844 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:13.846+0000 7f9d8a7a0640 1 --2- 172.21.15.142:0/3606920009 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9d8410cc50 0x7f9d8419de10 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:13.844 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:13.846+0000 7f9d8a7a0640 1 --2- 172.21.15.142:0/3606920009 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9d841040a0 0x7f9d84198a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:13.844 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:13.846+0000 7f9d8a7a0640 1 --2- 172.21.15.142:0/3606920009 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9d841036a0 0x7f9d84198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:13.844 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:13.846+0000 7f9d8a7a0640 1 -- 172.21.15.142:0/3606920009 >> 172.21.15.142:0/3606920009 conn(0x7f9d840fd2d0 msgr2=0x7f9d84104620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:29:13.844 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:13.846+0000 7f9d8a7a0640 1 -- 172.21.15.142:0/3606920009 shutdown_connections 2024-09-06T14:29:13.844 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:13.846+0000 7f9d8a7a0640 1 -- 172.21.15.142:0/3606920009 wait complete. 2024-09-06T14:29:14.621 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:29:14 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: ::ffff:172.21.15.162 - - [06/Sep/2024:14:29:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-06T14:29:14.959 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:29:14 smithi142 ceph-mon[37372]: pgmap v134: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:29:14.959 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:29:14 smithi142 ceph-mon[37372]: from='client.? 172.21.15.142:0/3606920009' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:29:14.960 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:29:14 smithi142 ceph-mon[37372]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-06T14:29:14.960 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:29:14 smithi142 ceph-mon[37372]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:29:14.960 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:29:14 smithi142 ceph-mon[37372]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-06T14:29:14.960 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:29:14 smithi142 ceph-mon[37372]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:29:14.960 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:29:14 smithi142 ceph-mon[29958]: pgmap v134: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:29:14.960 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:29:14 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/3606920009' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:29:14.960 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:29:14 smithi142 ceph-mon[29958]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-06T14:29:14.960 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:29:14 smithi142 ceph-mon[29958]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:29:14.960 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:29:14 smithi142 ceph-mon[29958]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-06T14:29:14.961 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:29:14 smithi142 ceph-mon[29958]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:29:15.048 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:29:14 smithi162 ceph-mon[33787]: pgmap v134: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:29:15.048 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:29:14 smithi162 ceph-mon[33787]: from='client.? 172.21.15.142:0/3606920009' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:29:15.048 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:29:14 smithi162 ceph-mon[33787]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-06T14:29:15.048 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:29:14 smithi162 ceph-mon[33787]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-06T14:29:15.048 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:29:14 smithi162 ceph-mon[33787]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-06T14:29:15.048 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:29:14 smithi162 ceph-mon[33787]: from='mgr.14601 ' entity='mgr.y' 2024-09-06T14:29:16.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:29:16 smithi142 ceph-mon[29958]: pgmap v135: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:29:16.960 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:29:16 smithi142 ceph-mon[37372]: pgmap v135: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:29:17.048 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:29:16 smithi162 ceph-mon[33787]: pgmap v135: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:29:17.547 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 06 14:29:17 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-iscsi-iscsi-a[59853]: debug there is no tcmu-runner data available 2024-09-06T14:29:17.648 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph health --format=json 2024-09-06T14:29:17.903 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:29:18.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:18.346+0000 7f340769a640 1 -- 172.21.15.142:0/501797200 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f34001084f0 msgr2=0x7f3400108930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:29:18.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:18.346+0000 7f340769a640 1 --2- 172.21.15.142:0/501797200 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f34001084f0 0x7f3400108930 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f33f0009860 tx=0x7f33f002a340 comp rx=0 tx=0).stop 2024-09-06T14:29:18.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:18.346+0000 7f340769a640 1 -- 172.21.15.142:0/501797200 shutdown_connections 2024-09-06T14:29:18.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:18.346+0000 7f340769a640 1 --2- 172.21.15.142:0/501797200 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3400108e70 0x7f340010f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:18.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:18.346+0000 7f340769a640 1 --2- 172.21.15.142:0/501797200 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f34001084f0 0x7f3400108930 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:18.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:18.346+0000 7f340769a640 1 --2- 172.21.15.142:0/501797200 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3400102ae0 0x7f3400102ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:18.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:18.346+0000 7f340769a640 1 -- 172.21.15.142:0/501797200 >> 172.21.15.142:0/501797200 conn(0x7f34000fe530 msgr2=0x7f3400100950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:29:18.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:18.347+0000 7f340769a640 1 -- 172.21.15.142:0/501797200 shutdown_connections 2024-09-06T14:29:18.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:18.347+0000 7f340769a640 1 -- 172.21.15.142:0/501797200 wait complete. 2024-09-06T14:29:18.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:18.347+0000 7f340769a640 1 Processor -- start 2024-09-06T14:29:18.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:18.347+0000 7f340769a640 1 -- start start 2024-09-06T14:29:18.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:18.347+0000 7f340769a640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3400102ae0 0x7f3400198770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:29:18.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:18.347+0000 7f340769a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f34001084f0 0x7f3400198cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:29:18.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:18.347+0000 7f340769a640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3400108e70 0x7f340019e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:29:18.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:18.347+0000 7f340769a640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f3400112240 con 0x7f3400102ae0 2024-09-06T14:29:18.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:18.347+0000 7f340769a640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f34001120c0 con 0x7f34001084f0 2024-09-06T14:29:18.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:18.347+0000 7f340769a640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7f34001123c0 con 0x7f3400108e70 2024-09-06T14:29:18.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:18.348+0000 7f3405c37640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3400108e70 0x7f340019e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:29:18.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:18.348+0000 7f3405436640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3400102ae0 0x7f3400198770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:29:18.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:18.348+0000 7f3405c37640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3400108e70 0x7f340019e050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:46952/0 (socket says 172.21.15.142:46952) 2024-09-06T14:29:18.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:18.348+0000 7f3405c37640 1 -- 172.21.15.142:0/487393331 learned_addr learned my addr 172.21.15.142:0/487393331 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:29:18.348 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:18.348+0000 7f3405436640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3400102ae0 0x7f3400198770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:32874/0 (socket says 172.21.15.142:32874) 2024-09-06T14:29:18.348 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:18.348+0000 7f3404c35640 1 --2- 172.21.15.142:0/487393331 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f34001084f0 0x7f3400198cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:29:18.348 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:18.348+0000 7f3405c37640 1 -- 172.21.15.142:0/487393331 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f34001084f0 msgr2=0x7f3400198cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:29:18.348 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:18.348+0000 7f3405c37640 1 --2- 172.21.15.142:0/487393331 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f34001084f0 0x7f3400198cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:18.348 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:18.348+0000 7f3405c37640 1 -- 172.21.15.142:0/487393331 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3400102ae0 msgr2=0x7f3400198770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:29:18.348 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:18.348+0000 7f3405c37640 1 --2- 172.21.15.142:0/487393331 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3400102ae0 0x7f3400198770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:18.348 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:18.348+0000 7f3405c37640 1 -- 172.21.15.142:0/487393331 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f340019e820 con 0x7f3400108e70 2024-09-06T14:29:18.348 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:18.348+0000 7f3405436640 1 --2- 172.21.15.142:0/487393331 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3400102ae0 0x7f3400198770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-06T14:29:18.348 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:18.348+0000 7f3405c37640 1 --2- 172.21.15.142:0/487393331 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3400108e70 0x7f340019e050 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f33f800efc0 tx=0x7f33f800c800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:29:18.348 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:18.348+0000 7f33f67fc640 1 -- 172.21.15.142:0/487393331 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33f8003970 con 0x7f3400108e70 2024-09-06T14:29:18.348 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:18.348+0000 7f340769a640 1 -- 172.21.15.142:0/487393331 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f340019eb10 con 0x7f3400108e70 2024-09-06T14:29:18.348 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:18.348+0000 7f33f67fc640 1 -- 172.21.15.142:0/487393331 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7f33f8004020 con 0x7f3400108e70 2024-09-06T14:29:18.348 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:18.348+0000 7f33f67fc640 1 -- 172.21.15.142:0/487393331 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33f800b830 con 0x7f3400108e70 2024-09-06T14:29:18.349 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:18.348+0000 7f340769a640 1 -- 172.21.15.142:0/487393331 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f34000750e0 con 0x7f3400108e70 2024-09-06T14:29:18.349 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:18.349+0000 7f340769a640 1 -- 172.21.15.142:0/487393331 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3400102ec0 con 0x7f3400108e70 2024-09-06T14:29:18.349 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:18.350+0000 7f33f67fc640 1 -- 172.21.15.142:0/487393331 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 22) ==== 99844+0+0 (secure 0 0 0) 0x7f33f801d020 con 0x7f3400108e70 2024-09-06T14:29:18.349 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:18.350+0000 7f33f67fc640 1 --2- 172.21.15.142:0/487393331 >> v2:172.21.15.142:6800/2944545612 conn(0x7f33ec077620 0x7f33ec079ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:29:18.350 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:18.351+0000 7f33f67fc640 1 -- 172.21.15.142:0/487393331 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(64..64 src has 1..64) ==== 6130+0+0 (secure 0 0 0) 0x7f33f8099670 con 0x7f3400108e70 2024-09-06T14:29:18.350 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:18.351+0000 7f3405436640 1 --2- 172.21.15.142:0/487393331 >> v2:172.21.15.142:6800/2944545612 conn(0x7f33ec077620 0x7f33ec079ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:29:18.350 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:18.351+0000 7f3405436640 1 --2- 172.21.15.142:0/487393331 >> v2:172.21.15.142:6800/2944545612 conn(0x7f33ec077620 0x7f33ec079ae0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f33fc009690 tx=0x7f33fc00f7a0 comp rx=0 tx=0).ready entity=mgr.14601 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:29:18.351 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:18.354+0000 7f33f67fc640 1 -- 172.21.15.142:0/487393331 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7f33f8066920 con 0x7f3400108e70 2024-09-06T14:29:18.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:18.540+0000 7f340769a640 1 -- 172.21.15.142:0/487393331 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f340010ac50 con 0x7f3400108e70 2024-09-06T14:29:18.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:18.540+0000 7f33f67fc640 1 -- 172.21.15.142:0/487393331 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+212 (secure 0 0 0) 0x7f33f806b340 con 0x7f3400108e70 2024-09-06T14:29:18.538 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:29:18.538 INFO:teuthology.orchestra.run.smithi142.stdout:{"status":"HEALTH_WARN","checks":{"MGR_MODULE_DEPENDENCY":{"severity":"HEALTH_WARN","summary":{"message":"Module 'volumes' has failed dependency: No module named 'ceph.fs'","count":1},"muted":false}},"mutes":[]} 2024-09-06T14:29:18.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:18.542+0000 7f340769a640 1 -- 172.21.15.142:0/487393331 >> v2:172.21.15.142:6800/2944545612 conn(0x7f33ec077620 msgr2=0x7f33ec079ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:29:18.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:18.542+0000 7f340769a640 1 --2- 172.21.15.142:0/487393331 >> v2:172.21.15.142:6800/2944545612 conn(0x7f33ec077620 0x7f33ec079ae0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f33fc009690 tx=0x7f33fc00f7a0 comp rx=0 tx=0).stop 2024-09-06T14:29:18.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:18.542+0000 7f340769a640 1 -- 172.21.15.142:0/487393331 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3400108e70 msgr2=0x7f340019e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:29:18.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:18.542+0000 7f340769a640 1 --2- 172.21.15.142:0/487393331 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3400108e70 0x7f340019e050 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f33f800efc0 tx=0x7f33f800c800 comp rx=0 tx=0).stop 2024-09-06T14:29:18.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:18.542+0000 7f340769a640 1 -- 172.21.15.142:0/487393331 shutdown_connections 2024-09-06T14:29:18.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:18.542+0000 7f340769a640 1 --2- 172.21.15.142:0/487393331 >> v2:172.21.15.142:6800/2944545612 conn(0x7f33ec077620 0x7f33ec079ae0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:18.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:18.542+0000 7f340769a640 1 --2- 172.21.15.142:0/487393331 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3400108e70 0x7f340019e050 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:18.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:18.542+0000 7f340769a640 1 --2- 172.21.15.142:0/487393331 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f34001084f0 0x7f3400198cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:18.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:18.542+0000 7f340769a640 1 --2- 172.21.15.142:0/487393331 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3400102ae0 0x7f3400198770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:18.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:18.542+0000 7f340769a640 1 -- 172.21.15.142:0/487393331 >> 172.21.15.142:0/487393331 conn(0x7f34000fe530 msgr2=0x7f3400100920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:29:18.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:18.542+0000 7f340769a640 1 -- 172.21.15.142:0/487393331 shutdown_connections 2024-09-06T14:29:18.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:18.543+0000 7f340769a640 1 -- 172.21.15.142:0/487393331 wait complete. 2024-09-06T14:29:18.959 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:29:18 smithi142 ceph-mon[37372]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-06T14:29:18.959 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:29:18 smithi142 ceph-mon[37372]: pgmap v136: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:29:18.959 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:29:18 smithi142 ceph-mon[37372]: from='client.? 172.21.15.142:0/487393331' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:29:18.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:29:18 smithi142 ceph-mon[29958]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-06T14:29:18.960 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:29:18 smithi142 ceph-mon[29958]: pgmap v136: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:29:18.960 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:29:18 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/487393331' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:29:19.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:29:18 smithi162 ceph-mon[33787]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-06T14:29:19.048 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:29:18 smithi162 ceph-mon[33787]: pgmap v136: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:29:19.048 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:29:18 smithi162 ceph-mon[33787]: from='client.? 172.21.15.142:0/487393331' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:29:20.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:29:20 smithi142 ceph-mon[29958]: pgmap v137: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:29:20.959 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:29:20 smithi142 ceph-mon[37372]: pgmap v137: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:29:21.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:29:20 smithi162 ceph-mon[33787]: pgmap v137: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:29:22.298 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph health --format=json 2024-09-06T14:29:22.558 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:29:22.930 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:29:22 smithi142 ceph-mon[29958]: pgmap v138: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:29:22.930 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:29:22 smithi142 ceph-mon[37372]: pgmap v138: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:29:23.047 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:23.011+0000 7fa87597a640 1 -- 172.21.15.142:0/701766468 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa870107410 msgr2=0x7fa870107850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:29:23.047 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:23.011+0000 7fa87597a640 1 --2- 172.21.15.142:0/701766468 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa870107410 0x7fa870107850 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fa864009860 tx=0x7fa86402a340 comp rx=0 tx=0).stop 2024-09-06T14:29:23.047 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:23.011+0000 7fa87597a640 1 -- 172.21.15.142:0/701766468 shutdown_connections 2024-09-06T14:29:23.047 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:23.011+0000 7fa87597a640 1 --2- 172.21.15.142:0/701766468 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa870107d90 0x7fa87010e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:23.048 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:23.011+0000 7fa87597a640 1 --2- 172.21.15.142:0/701766468 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa870107410 0x7fa870107850 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:23.048 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:23.011+0000 7fa87597a640 1 --2- 172.21.15.142:0/701766468 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa870101a00 0x7fa870101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:23.048 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:23.011+0000 7fa87597a640 1 -- 172.21.15.142:0/701766468 >> 172.21.15.142:0/701766468 conn(0x7fa8700fd470 msgr2=0x7fa8700ff890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:29:23.048 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:23.011+0000 7fa87597a640 1 -- 172.21.15.142:0/701766468 shutdown_connections 2024-09-06T14:29:23.048 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:23.011+0000 7fa87597a640 1 -- 172.21.15.142:0/701766468 wait complete. 2024-09-06T14:29:23.048 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:23.011+0000 7fa87597a640 1 Processor -- start 2024-09-06T14:29:23.048 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:23.011+0000 7fa87597a640 1 -- start start 2024-09-06T14:29:23.048 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:23.012+0000 7fa87597a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa870101a00 0x7fa87019a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:29:23.048 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:23.012+0000 7fa87597a640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa870107410 0x7fa87019ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:29:23.048 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:23.012+0000 7fa87597a640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa870107d90 0x7fa8701a0230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:29:23.048 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:23.012+0000 7fa87597a640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fa870111160 con 0x7fa870107d90 2024-09-06T14:29:23.048 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:23.012+0000 7fa87597a640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa870110fe0 con 0x7fa870101a00 2024-09-06T14:29:23.049 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:23.012+0000 7fa87597a640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7fa8701112e0 con 0x7fa870107410 2024-09-06T14:29:23.049 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:23.012+0000 7fa86e7fc640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa870107410 0x7fa87019ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:29:23.049 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:23.012+0000 7fa86f7fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa870107d90 0x7fa8701a0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:29:23.049 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:23.012+0000 7fa86e7fc640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa870107410 0x7fa87019ae90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:46964/0 (socket says 172.21.15.142:46964) 2024-09-06T14:29:23.049 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:23.012+0000 7fa86f7fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa870107d90 0x7fa8701a0230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:32904/0 (socket says 172.21.15.142:32904) 2024-09-06T14:29:23.049 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:23.012+0000 7fa86e7fc640 1 -- 172.21.15.142:0/2965210771 learned_addr learned my addr 172.21.15.142:0/2965210771 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:29:23.049 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:23.012+0000 7fa86e7fc640 1 -- 172.21.15.142:0/2965210771 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa870101a00 msgr2=0x7fa87019a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:29:23.049 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:23.012+0000 7fa86effd640 1 --2- 172.21.15.142:0/2965210771 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa870101a00 0x7fa87019a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:29:23.049 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:23.012+0000 7fa86e7fc640 1 --2- 172.21.15.142:0/2965210771 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa870101a00 0x7fa87019a950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:23.049 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:23.012+0000 7fa86e7fc640 1 -- 172.21.15.142:0/2965210771 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa870107d90 msgr2=0x7fa8701a0230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:29:23.049 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:23.012+0000 7fa86e7fc640 1 --2- 172.21.15.142:0/2965210771 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa870107d90 0x7fa8701a0230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:23.049 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:23.012+0000 7fa86e7fc640 1 -- 172.21.15.142:0/2965210771 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8701a0a00 con 0x7fa870107410 2024-09-06T14:29:23.049 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:23.012+0000 7fa86f7fe640 1 --2- 172.21.15.142:0/2965210771 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa870107d90 0x7fa8701a0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-06T14:29:23.049 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:23.012+0000 7fa86e7fc640 1 --2- 172.21.15.142:0/2965210771 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa870107410 0x7fa87019ae90 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fa864035040 tx=0x7fa864033cb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:29:23.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:23.012+0000 7fa874978640 1 -- 172.21.15.142:0/2965210771 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa864037040 con 0x7fa870107410 2024-09-06T14:29:23.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:23.012+0000 7fa87597a640 1 -- 172.21.15.142:0/2965210771 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8701a0c90 con 0x7fa870107410 2024-09-06T14:29:23.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:23.012+0000 7fa874978640 1 -- 172.21.15.142:0/2965210771 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7fa864040070 con 0x7fa870107410 2024-09-06T14:29:23.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:23.013+0000 7fa874978640 1 -- 172.21.15.142:0/2965210771 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa864003830 con 0x7fa870107410 2024-09-06T14:29:23.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:23.013+0000 7fa87597a640 1 -- 172.21.15.142:0/2965210771 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8701a73b0 con 0x7fa870107410 2024-09-06T14:29:23.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:23.013+0000 7fa87597a640 1 -- 172.21.15.142:0/2965210771 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa870101de0 con 0x7fa870107410 2024-09-06T14:29:23.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:23.014+0000 7fa874978640 1 -- 172.21.15.142:0/2965210771 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 22) ==== 99844+0+0 (secure 0 0 0) 0x7fa864003a90 con 0x7fa870107410 2024-09-06T14:29:23.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:23.015+0000 7fa874978640 1 --2- 172.21.15.142:0/2965210771 >> v2:172.21.15.142:6800/2944545612 conn(0x7fa858077620 0x7fa858079ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:29:23.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:23.015+0000 7fa874978640 1 -- 172.21.15.142:0/2965210771 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(64..64 src has 1..64) ==== 6130+0+0 (secure 0 0 0) 0x7fa8640b9910 con 0x7fa870107410 2024-09-06T14:29:23.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:23.015+0000 7fa86effd640 1 --2- 172.21.15.142:0/2965210771 >> v2:172.21.15.142:6800/2944545612 conn(0x7fa858077620 0x7fa858079ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:29:23.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:23.015+0000 7fa86effd640 1 --2- 172.21.15.142:0/2965210771 >> v2:172.21.15.142:6800/2944545612 conn(0x7fa858077620 0x7fa858079ae0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa86000a810 tx=0x7fa860008040 comp rx=0 tx=0).ready entity=mgr.14601 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:29:23.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:23.019+0000 7fa874978640 1 -- 172.21.15.142:0/2965210771 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7fa8640829d0 con 0x7fa870107410 2024-09-06T14:29:23.051 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:29:22 smithi162 ceph-mon[33787]: pgmap v138: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:29:23.203 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:23.206+0000 7fa87597a640 1 -- 172.21.15.142:0/2965210771 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fa8701a76d0 con 0x7fa870107410 2024-09-06T14:29:23.204 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:23.206+0000 7fa874978640 1 -- 172.21.15.142:0/2965210771 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+212 (secure 0 0 0) 0x7fa864086b40 con 0x7fa870107410 2024-09-06T14:29:23.204 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:29:23.204 INFO:teuthology.orchestra.run.smithi142.stdout:{"status":"HEALTH_WARN","checks":{"MGR_MODULE_DEPENDENCY":{"severity":"HEALTH_WARN","summary":{"message":"Module 'volumes' has failed dependency: No module named 'ceph.fs'","count":1},"muted":false}},"mutes":[]} 2024-09-06T14:29:23.206 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:23.208+0000 7fa87597a640 1 -- 172.21.15.142:0/2965210771 >> v2:172.21.15.142:6800/2944545612 conn(0x7fa858077620 msgr2=0x7fa858079ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:29:23.206 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:23.208+0000 7fa87597a640 1 --2- 172.21.15.142:0/2965210771 >> v2:172.21.15.142:6800/2944545612 conn(0x7fa858077620 0x7fa858079ae0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa86000a810 tx=0x7fa860008040 comp rx=0 tx=0).stop 2024-09-06T14:29:23.206 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:23.208+0000 7fa87597a640 1 -- 172.21.15.142:0/2965210771 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa870107410 msgr2=0x7fa87019ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:29:23.206 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:23.208+0000 7fa87597a640 1 --2- 172.21.15.142:0/2965210771 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa870107410 0x7fa87019ae90 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fa864035040 tx=0x7fa864033cb0 comp rx=0 tx=0).stop 2024-09-06T14:29:23.206 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:23.208+0000 7fa87597a640 1 -- 172.21.15.142:0/2965210771 shutdown_connections 2024-09-06T14:29:23.206 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:23.208+0000 7fa87597a640 1 --2- 172.21.15.142:0/2965210771 >> v2:172.21.15.142:6800/2944545612 conn(0x7fa858077620 0x7fa858079ae0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:23.206 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:23.208+0000 7fa87597a640 1 --2- 172.21.15.142:0/2965210771 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa870107d90 0x7fa8701a0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:23.206 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:23.208+0000 7fa87597a640 1 --2- 172.21.15.142:0/2965210771 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa870107410 0x7fa87019ae90 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:23.206 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:23.208+0000 7fa87597a640 1 --2- 172.21.15.142:0/2965210771 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa870101a00 0x7fa87019a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:23.206 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:23.208+0000 7fa87597a640 1 -- 172.21.15.142:0/2965210771 >> 172.21.15.142:0/2965210771 conn(0x7fa8700fd470 msgr2=0x7fa87006b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:29:23.206 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:23.209+0000 7fa87597a640 1 -- 172.21.15.142:0/2965210771 shutdown_connections 2024-09-06T14:29:23.207 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:23.209+0000 7fa87597a640 1 -- 172.21.15.142:0/2965210771 wait complete. 2024-09-06T14:29:23.951 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:29:23 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/2965210771' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:29:23.951 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:29:23 smithi142 ceph-mon[37372]: from='client.? 172.21.15.142:0/2965210771' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:29:24.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:29:23 smithi162 ceph-mon[33787]: from='client.? 172.21.15.142:0/2965210771' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:29:24.709 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:29:24 smithi142 ceph-mon[29958]: pgmap v139: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:29:24.710 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:29:24 smithi142 ceph-mon[37372]: pgmap v139: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:29:24.710 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:29:24 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: ::ffff:172.21.15.162 - - [06/Sep/2024:14:29:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-06T14:29:25.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:29:24 smithi162 ceph-mon[33787]: pgmap v139: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:29:26.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:29:26 smithi142 ceph-mon[29958]: pgmap v140: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:29:26.960 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:29:26 smithi142 ceph-mon[29958]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-06T14:29:26.960 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:29:26 smithi142 ceph-mon[37372]: pgmap v140: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:29:26.960 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:29:26 smithi142 ceph-mon[37372]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-06T14:29:26.979 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph health --format=json 2024-09-06T14:29:27.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:29:26 smithi162 ceph-mon[33787]: pgmap v140: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:29:27.048 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:29:26 smithi162 ceph-mon[33787]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-06T14:29:27.244 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:29:27.547 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 06 14:29:27 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-iscsi-iscsi-a[59853]: debug there is no tcmu-runner data available 2024-09-06T14:29:27.721 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:27.722+0000 7f4213abc640 1 -- 172.21.15.142:0/3339964931 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f420c108510 msgr2=0x7f420c108950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:29:27.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:27.722+0000 7f4213abc640 1 --2- 172.21.15.142:0/3339964931 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f420c108510 0x7f420c108950 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f4208009900 tx=0x7f420802a410 comp rx=0 tx=0).stop 2024-09-06T14:29:27.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:27.722+0000 7f4213abc640 1 -- 172.21.15.142:0/3339964931 shutdown_connections 2024-09-06T14:29:27.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:27.722+0000 7f4213abc640 1 --2- 172.21.15.142:0/3339964931 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f420c108e90 0x7f420c10f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:27.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:27.722+0000 7f4213abc640 1 --2- 172.21.15.142:0/3339964931 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f420c108510 0x7f420c108950 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:27.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:27.722+0000 7f4213abc640 1 --2- 172.21.15.142:0/3339964931 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f420c102b00 0x7f420c102ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:27.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:27.722+0000 7f4213abc640 1 -- 172.21.15.142:0/3339964931 >> 172.21.15.142:0/3339964931 conn(0x7f420c0fe570 msgr2=0x7f420c100990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:29:27.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:27.722+0000 7f4213abc640 1 -- 172.21.15.142:0/3339964931 shutdown_connections 2024-09-06T14:29:27.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:27.722+0000 7f4213abc640 1 -- 172.21.15.142:0/3339964931 wait complete. 2024-09-06T14:29:27.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:27.722+0000 7f4213abc640 1 Processor -- start 2024-09-06T14:29:27.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:27.722+0000 7f4213abc640 1 -- start start 2024-09-06T14:29:27.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:27.723+0000 7f4213abc640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f420c102b00 0x7f420c198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:29:27.723 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:27.723+0000 7f4213abc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f420c108510 0x7f420c198c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:29:27.723 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:27.723+0000 7f4213abc640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f420c108e90 0x7f420c19e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:29:27.723 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:27.723+0000 7f4213abc640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f420c112260 con 0x7f420c102b00 2024-09-06T14:29:27.723 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:27.723+0000 7f4213abc640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f420c1120e0 con 0x7f420c108510 2024-09-06T14:29:27.723 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:27.723+0000 7f4213abc640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7f420c1123e0 con 0x7f420c108e90 2024-09-06T14:29:27.723 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:27.723+0000 7f4211858640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f420c102b00 0x7f420c198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:29:27.723 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:27.723+0000 7f4212059640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f420c108e90 0x7f420c19e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:29:27.723 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:27.723+0000 7f4211858640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f420c102b00 0x7f420c198750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:58130/0 (socket says 172.21.15.142:58130) 2024-09-06T14:29:27.723 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:27.723+0000 7f4211858640 1 -- 172.21.15.142:0/1066673126 learned_addr learned my addr 172.21.15.142:0/1066673126 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:29:27.723 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:27.723+0000 7f4211057640 1 --2- 172.21.15.142:0/1066673126 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f420c108510 0x7f420c198c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:29:27.723 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:27.723+0000 7f4211858640 1 -- 172.21.15.142:0/1066673126 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f420c108e90 msgr2=0x7f420c19e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:29:27.724 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:27.723+0000 7f4211858640 1 --2- 172.21.15.142:0/1066673126 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f420c108e90 0x7f420c19e030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:27.724 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:27.723+0000 7f4211858640 1 -- 172.21.15.142:0/1066673126 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f420c108510 msgr2=0x7f420c198c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:29:27.724 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:27.723+0000 7f4211858640 1 --2- 172.21.15.142:0/1066673126 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f420c108510 0x7f420c198c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:27.724 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:27.723+0000 7f4211858640 1 -- 172.21.15.142:0/1066673126 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f420c19e800 con 0x7f420c102b00 2024-09-06T14:29:27.724 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:27.723+0000 7f4211057640 1 --2- 172.21.15.142:0/1066673126 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f420c108510 0x7f420c198c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-06T14:29:27.724 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:27.723+0000 7f4211858640 1 --2- 172.21.15.142:0/1066673126 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f420c102b00 0x7f420c198750 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f41fc00b4a0 tx=0x7f41fc00b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:29:27.724 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:27.723+0000 7f4202ffd640 1 -- 172.21.15.142:0/1066673126 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41fc015020 con 0x7f420c102b00 2024-09-06T14:29:27.724 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:27.723+0000 7f4202ffd640 1 -- 172.21.15.142:0/1066673126 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7f41fc011cb0 con 0x7f420c102b00 2024-09-06T14:29:27.724 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:27.723+0000 7f4213abc640 1 -- 172.21.15.142:0/1066673126 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f420c19eaf0 con 0x7f420c102b00 2024-09-06T14:29:27.724 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:27.724+0000 7f4202ffd640 1 -- 172.21.15.142:0/1066673126 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41fc010cd0 con 0x7f420c102b00 2024-09-06T14:29:27.724 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:27.724+0000 7f4213abc640 1 -- 172.21.15.142:0/1066673126 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f420c1a5380 con 0x7f420c102b00 2024-09-06T14:29:27.725 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:27.724+0000 7f4213abc640 1 -- 172.21.15.142:0/1066673126 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f420c102ee0 con 0x7f420c102b00 2024-09-06T14:29:27.725 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:27.725+0000 7f4202ffd640 1 -- 172.21.15.142:0/1066673126 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 22) ==== 99844+0+0 (secure 0 0 0) 0x7f41fc018020 con 0x7f420c102b00 2024-09-06T14:29:27.725 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:27.726+0000 7f4202ffd640 1 --2- 172.21.15.142:0/1066673126 >> v2:172.21.15.142:6800/2944545612 conn(0x7f41f8077620 0x7f41f8079ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:29:27.725 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:27.726+0000 7f4202ffd640 1 -- 172.21.15.142:0/1066673126 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 6130+0+0 (secure 0 0 0) 0x7f41fc0990b0 con 0x7f420c102b00 2024-09-06T14:29:27.725 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:27.726+0000 7f4211057640 1 --2- 172.21.15.142:0/1066673126 >> v2:172.21.15.142:6800/2944545612 conn(0x7f41f8077620 0x7f41f8079ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:29:27.726 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:27.726+0000 7f4211057640 1 --2- 172.21.15.142:0/1066673126 >> v2:172.21.15.142:6800/2944545612 conn(0x7f41f8077620 0x7f41f8079ae0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4208000c00 tx=0x7f4208031400 comp rx=0 tx=0).ready entity=mgr.14601 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:29:27.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:27.730+0000 7f4202ffd640 1 -- 172.21.15.142:0/1066673126 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7f41fc01f020 con 0x7f420c102b00 2024-09-06T14:29:27.914 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:27.917+0000 7f4213abc640 1 -- 172.21.15.142:0/1066673126 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f420c10ac70 con 0x7f420c102b00 2024-09-06T14:29:27.915 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:27.917+0000 7f4202ffd640 1 -- 172.21.15.142:0/1066673126 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+212 (secure 0 0 0) 0x7f41fc0621f0 con 0x7f420c102b00 2024-09-06T14:29:27.915 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:29:27.915 INFO:teuthology.orchestra.run.smithi142.stdout:{"status":"HEALTH_WARN","checks":{"MGR_MODULE_DEPENDENCY":{"severity":"HEALTH_WARN","summary":{"message":"Module 'volumes' has failed dependency: No module named 'ceph.fs'","count":1},"muted":false}},"mutes":[]} 2024-09-06T14:29:27.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:27.919+0000 7f4213abc640 1 -- 172.21.15.142:0/1066673126 >> v2:172.21.15.142:6800/2944545612 conn(0x7f41f8077620 msgr2=0x7f41f8079ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:29:27.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:27.919+0000 7f4213abc640 1 --2- 172.21.15.142:0/1066673126 >> v2:172.21.15.142:6800/2944545612 conn(0x7f41f8077620 0x7f41f8079ae0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4208000c00 tx=0x7f4208031400 comp rx=0 tx=0).stop 2024-09-06T14:29:27.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:27.919+0000 7f4213abc640 1 -- 172.21.15.142:0/1066673126 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f420c102b00 msgr2=0x7f420c198750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:29:27.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:27.919+0000 7f4213abc640 1 --2- 172.21.15.142:0/1066673126 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f420c102b00 0x7f420c198750 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f41fc00b4a0 tx=0x7f41fc00b8a0 comp rx=0 tx=0).stop 2024-09-06T14:29:27.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:27.920+0000 7f4213abc640 1 -- 172.21.15.142:0/1066673126 shutdown_connections 2024-09-06T14:29:27.918 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:27.920+0000 7f4213abc640 1 --2- 172.21.15.142:0/1066673126 >> v2:172.21.15.142:6800/2944545612 conn(0x7f41f8077620 0x7f41f8079ae0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:27.918 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:27.920+0000 7f4213abc640 1 --2- 172.21.15.142:0/1066673126 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f420c108e90 0x7f420c19e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:27.918 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:27.920+0000 7f4213abc640 1 --2- 172.21.15.142:0/1066673126 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f420c108510 0x7f420c198c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:27.918 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:27.920+0000 7f4213abc640 1 --2- 172.21.15.142:0/1066673126 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f420c102b00 0x7f420c198750 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:27.918 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:27.920+0000 7f4213abc640 1 -- 172.21.15.142:0/1066673126 >> 172.21.15.142:0/1066673126 conn(0x7f420c0fe570 msgr2=0x7f420c100960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:29:27.918 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:27.920+0000 7f4213abc640 1 -- 172.21.15.142:0/1066673126 shutdown_connections 2024-09-06T14:29:27.918 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:27.920+0000 7f4213abc640 1 -- 172.21.15.142:0/1066673126 wait complete. 2024-09-06T14:29:28.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:29:28 smithi142 ceph-mon[29958]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-06T14:29:28.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:29:28 smithi142 ceph-mon[29958]: pgmap v141: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:29:28.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:29:28 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/1066673126' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:29:28.960 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:29:28 smithi142 ceph-mon[37372]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-06T14:29:28.960 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:29:28 smithi142 ceph-mon[37372]: pgmap v141: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:29:28.960 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:29:28 smithi142 ceph-mon[37372]: from='client.? 172.21.15.142:0/1066673126' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:29:29.048 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:29:28 smithi162 ceph-mon[33787]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-06T14:29:29.048 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:29:28 smithi162 ceph-mon[33787]: pgmap v141: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:29:29.048 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:29:28 smithi162 ceph-mon[33787]: from='client.? 172.21.15.142:0/1066673126' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:29:30.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:29:30 smithi142 ceph-mon[29958]: pgmap v142: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:29:30.959 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:29:30 smithi142 ceph-mon[37372]: pgmap v142: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:29:31.048 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:29:30 smithi162 ceph-mon[33787]: pgmap v142: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:29:31.553 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph health --format=json 2024-09-06T14:29:31.815 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:29:32.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:32.301+0000 7f6fe023c640 1 -- 172.21.15.142:0/2838538678 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6fd81084f0 msgr2=0x7f6fd8108930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:29:32.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:32.301+0000 7f6fe023c640 1 --2- 172.21.15.142:0/2838538678 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6fd81084f0 0x7f6fd8108930 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f6fc8009860 tx=0x7f6fc802a340 comp rx=0 tx=0).stop 2024-09-06T14:29:32.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:32.301+0000 7f6fe023c640 1 -- 172.21.15.142:0/2838538678 shutdown_connections 2024-09-06T14:29:32.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:32.301+0000 7f6fe023c640 1 --2- 172.21.15.142:0/2838538678 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6fd8108e70 0x7f6fd810f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:32.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:32.301+0000 7f6fe023c640 1 --2- 172.21.15.142:0/2838538678 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6fd81084f0 0x7f6fd8108930 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:32.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:32.301+0000 7f6fe023c640 1 --2- 172.21.15.142:0/2838538678 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6fd8102ae0 0x7f6fd8102ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:32.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:32.301+0000 7f6fe023c640 1 -- 172.21.15.142:0/2838538678 >> 172.21.15.142:0/2838538678 conn(0x7f6fd80fe530 msgr2=0x7f6fd8100950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:29:32.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:32.301+0000 7f6fe023c640 1 -- 172.21.15.142:0/2838538678 shutdown_connections 2024-09-06T14:29:32.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:32.301+0000 7f6fe023c640 1 -- 172.21.15.142:0/2838538678 wait complete. 2024-09-06T14:29:32.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:32.301+0000 7f6fe023c640 1 Processor -- start 2024-09-06T14:29:32.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:32.301+0000 7f6fe023c640 1 -- start start 2024-09-06T14:29:32.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:32.302+0000 7f6fe023c640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6fd8102ae0 0x7f6fd8198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:29:32.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:32.302+0000 7f6fe023c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6fd81084f0 0x7f6fd8198c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:29:32.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:32.302+0000 7f6fe023c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6fd8108e70 0x7f6fd819e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:29:32.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:32.302+0000 7f6fe023c640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f6fd8112240 con 0x7f6fd81084f0 2024-09-06T14:29:32.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:32.302+0000 7f6fe023c640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6fd81120c0 con 0x7f6fd8108e70 2024-09-06T14:29:32.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:32.302+0000 7f6fe023c640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7f6fd81123c0 con 0x7f6fd8102ae0 2024-09-06T14:29:32.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:32.302+0000 7f6fddfd8640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6fd8102ae0 0x7f6fd8198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:29:32.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:32.302+0000 7f6fdd7d7640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6fd81084f0 0x7f6fd8198c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:29:32.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:32.302+0000 7f6fdd7d7640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6fd81084f0 0x7f6fd8198c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:58152/0 (socket says 172.21.15.142:58152) 2024-09-06T14:29:32.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:32.302+0000 7f6fdd7d7640 1 -- 172.21.15.142:0/1222020293 learned_addr learned my addr 172.21.15.142:0/1222020293 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:29:32.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:32.302+0000 7f6fddfd8640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6fd8102ae0 0x7f6fd8198730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:42180/0 (socket says 172.21.15.142:42180) 2024-09-06T14:29:32.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:32.302+0000 7f6fde7d9640 1 --2- 172.21.15.142:0/1222020293 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6fd8108e70 0x7f6fd819e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:29:32.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:32.302+0000 7f6fdd7d7640 1 -- 172.21.15.142:0/1222020293 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6fd8102ae0 msgr2=0x7f6fd8198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:29:32.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:32.302+0000 7f6fdd7d7640 1 --2- 172.21.15.142:0/1222020293 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6fd8102ae0 0x7f6fd8198730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:32.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:32.302+0000 7f6fdd7d7640 1 -- 172.21.15.142:0/1222020293 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6fd8108e70 msgr2=0x7f6fd819e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:29:32.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:32.302+0000 7f6fdd7d7640 1 --2- 172.21.15.142:0/1222020293 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6fd8108e70 0x7f6fd819e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:32.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:32.302+0000 7f6fdd7d7640 1 -- 172.21.15.142:0/1222020293 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fd819e7e0 con 0x7f6fd81084f0 2024-09-06T14:29:32.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:32.302+0000 7f6fde7d9640 1 --2- 172.21.15.142:0/1222020293 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6fd8108e70 0x7f6fd819e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-06T14:29:32.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:32.302+0000 7f6fddfd8640 1 --2- 172.21.15.142:0/1222020293 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6fd8102ae0 0x7f6fd8198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-06T14:29:32.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:32.302+0000 7f6fdd7d7640 1 --2- 172.21.15.142:0/1222020293 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6fd81084f0 0x7f6fd8198c70 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f6fc8009980 tx=0x7f6fc8033d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:29:32.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:32.302+0000 7f6fceffd640 1 -- 172.21.15.142:0/1222020293 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fc803f020 con 0x7f6fd81084f0 2024-09-06T14:29:32.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:32.302+0000 7f6fe023c640 1 -- 172.21.15.142:0/1222020293 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6fd819ea70 con 0x7f6fd81084f0 2024-09-06T14:29:32.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:32.303+0000 7f6fceffd640 1 -- 172.21.15.142:0/1222020293 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7f6fc80044c0 con 0x7f6fd81084f0 2024-09-06T14:29:32.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:32.303+0000 7f6fceffd640 1 -- 172.21.15.142:0/1222020293 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fc8037040 con 0x7f6fd81084f0 2024-09-06T14:29:32.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:32.303+0000 7f6fe023c640 1 -- 172.21.15.142:0/1222020293 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6fd81a5190 con 0x7f6fd81084f0 2024-09-06T14:29:32.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:32.304+0000 7f6fe023c640 1 -- 172.21.15.142:0/1222020293 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6fd8102ec0 con 0x7f6fd81084f0 2024-09-06T14:29:32.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:32.304+0000 7f6fceffd640 1 -- 172.21.15.142:0/1222020293 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 22) ==== 99844+0+0 (secure 0 0 0) 0x7f6fc8042020 con 0x7f6fd81084f0 2024-09-06T14:29:32.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:32.304+0000 7f6fceffd640 1 --2- 172.21.15.142:0/1222020293 >> v2:172.21.15.142:6800/2944545612 conn(0x7f6fc4077620 0x7f6fc4079ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:29:32.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:32.305+0000 7f6fddfd8640 1 --2- 172.21.15.142:0/1222020293 >> v2:172.21.15.142:6800/2944545612 conn(0x7f6fc4077620 0x7f6fc4079ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:29:32.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:32.305+0000 7f6fceffd640 1 -- 172.21.15.142:0/1222020293 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 6130+0+0 (secure 0 0 0) 0x7f6fc80b9910 con 0x7f6fd81084f0 2024-09-06T14:29:32.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:32.305+0000 7f6fddfd8640 1 --2- 172.21.15.142:0/1222020293 >> v2:172.21.15.142:6800/2944545612 conn(0x7f6fc4077620 0x7f6fc4079ae0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6fd4005c40 tx=0x7f6fd400a590 comp rx=0 tx=0).ready entity=mgr.14601 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:29:32.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:32.308+0000 7f6fceffd640 1 -- 172.21.15.142:0/1222020293 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7f6fc8049020 con 0x7f6fd81084f0 2024-09-06T14:29:32.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:32.498+0000 7f6fe023c640 1 -- 172.21.15.142:0/1222020293 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f6fd810ac50 con 0x7f6fd81084f0 2024-09-06T14:29:32.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:32.498+0000 7f6fceffd640 1 -- 172.21.15.142:0/1222020293 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+212 (secure 0 0 0) 0x7f6fc80829d0 con 0x7f6fd81084f0 2024-09-06T14:29:32.496 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:29:32.496 INFO:teuthology.orchestra.run.smithi142.stdout:{"status":"HEALTH_WARN","checks":{"MGR_MODULE_DEPENDENCY":{"severity":"HEALTH_WARN","summary":{"message":"Module 'volumes' has failed dependency: No module named 'ceph.fs'","count":1},"muted":false}},"mutes":[]} 2024-09-06T14:29:32.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:32.500+0000 7f6fe023c640 1 -- 172.21.15.142:0/1222020293 >> v2:172.21.15.142:6800/2944545612 conn(0x7f6fc4077620 msgr2=0x7f6fc4079ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:29:32.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:32.500+0000 7f6fe023c640 1 --2- 172.21.15.142:0/1222020293 >> v2:172.21.15.142:6800/2944545612 conn(0x7f6fc4077620 0x7f6fc4079ae0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6fd4005c40 tx=0x7f6fd400a590 comp rx=0 tx=0).stop 2024-09-06T14:29:32.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:32.500+0000 7f6fe023c640 1 -- 172.21.15.142:0/1222020293 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6fd81084f0 msgr2=0x7f6fd8198c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:29:32.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:32.500+0000 7f6fe023c640 1 --2- 172.21.15.142:0/1222020293 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6fd81084f0 0x7f6fd8198c70 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f6fc8009980 tx=0x7f6fc8033d30 comp rx=0 tx=0).stop 2024-09-06T14:29:32.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:32.501+0000 7f6fe023c640 1 -- 172.21.15.142:0/1222020293 shutdown_connections 2024-09-06T14:29:32.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:32.501+0000 7f6fe023c640 1 --2- 172.21.15.142:0/1222020293 >> v2:172.21.15.142:6800/2944545612 conn(0x7f6fc4077620 0x7f6fc4079ae0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:32.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:32.501+0000 7f6fe023c640 1 --2- 172.21.15.142:0/1222020293 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6fd8108e70 0x7f6fd819e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:32.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:32.501+0000 7f6fe023c640 1 --2- 172.21.15.142:0/1222020293 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6fd81084f0 0x7f6fd8198c70 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:32.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:32.501+0000 7f6fe023c640 1 --2- 172.21.15.142:0/1222020293 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6fd8102ae0 0x7f6fd8198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:32.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:32.501+0000 7f6fe023c640 1 -- 172.21.15.142:0/1222020293 >> 172.21.15.142:0/1222020293 conn(0x7f6fd80fe530 msgr2=0x7f6fd8100920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:29:32.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:32.501+0000 7f6fe023c640 1 -- 172.21.15.142:0/1222020293 shutdown_connections 2024-09-06T14:29:32.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:32.501+0000 7f6fe023c640 1 -- 172.21.15.142:0/1222020293 wait complete. 2024-09-06T14:29:32.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:29:32 smithi142 ceph-mon[29958]: pgmap v143: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:29:32.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:29:32 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/1222020293' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:29:32.960 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:29:32 smithi142 ceph-mon[37372]: pgmap v143: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:29:32.960 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:29:32 smithi142 ceph-mon[37372]: from='client.? 172.21.15.142:0/1222020293' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:29:33.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:29:32 smithi162 ceph-mon[33787]: pgmap v143: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:29:33.048 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:29:32 smithi162 ceph-mon[33787]: from='client.? 172.21.15.142:0/1222020293' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:29:34.709 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:29:34 smithi142 ceph-mon[37372]: pgmap v144: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:29:34.710 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:29:34 smithi142 ceph-mon[29958]: pgmap v144: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:29:34.710 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:29:34 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: ::ffff:172.21.15.162 - - [06/Sep/2024:14:29:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-06T14:29:35.048 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:29:34 smithi162 ceph-mon[33787]: pgmap v144: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:29:36.109 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph health --format=json 2024-09-06T14:29:36.367 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:29:36.788 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:29:36 smithi142 ceph-mon[29958]: pgmap v145: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:29:36.788 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:29:36 smithi142 ceph-mon[37372]: pgmap v145: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:29:36.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:36.869+0000 7ff2db7f8640 1 -- 172.21.15.142:0/1447329123 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff2d4108e70 msgr2=0x7ff2d410f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:29:36.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:36.869+0000 7ff2db7f8640 1 --2- 172.21.15.142:0/1447329123 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff2d4108e70 0x7ff2d410f670 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7ff2cc009ca0 tx=0x7ff2cc02a8c0 comp rx=0 tx=0).stop 2024-09-06T14:29:36.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:36.869+0000 7ff2db7f8640 1 -- 172.21.15.142:0/1447329123 shutdown_connections 2024-09-06T14:29:36.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:36.869+0000 7ff2db7f8640 1 --2- 172.21.15.142:0/1447329123 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff2d4108e70 0x7ff2d410f670 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:36.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:36.869+0000 7ff2db7f8640 1 --2- 172.21.15.142:0/1447329123 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff2d41084f0 0x7ff2d4108930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:36.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:36.869+0000 7ff2db7f8640 1 --2- 172.21.15.142:0/1447329123 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff2d4102ae0 0x7ff2d4102ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:36.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:36.869+0000 7ff2db7f8640 1 -- 172.21.15.142:0/1447329123 >> 172.21.15.142:0/1447329123 conn(0x7ff2d40fe530 msgr2=0x7ff2d4100950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:29:36.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:36.869+0000 7ff2db7f8640 1 -- 172.21.15.142:0/1447329123 shutdown_connections 2024-09-06T14:29:36.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:36.869+0000 7ff2db7f8640 1 -- 172.21.15.142:0/1447329123 wait complete. 2024-09-06T14:29:36.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:36.869+0000 7ff2db7f8640 1 Processor -- start 2024-09-06T14:29:36.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:36.869+0000 7ff2db7f8640 1 -- start start 2024-09-06T14:29:36.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:36.869+0000 7ff2db7f8640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff2d4102ae0 0x7ff2d4198650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:29:36.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:36.869+0000 7ff2db7f8640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff2d41084f0 0x7ff2d4198b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:29:36.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:36.869+0000 7ff2db7f8640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff2d4108e70 0x7ff2d419df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:29:36.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:36.869+0000 7ff2db7f8640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7ff2d4112200 con 0x7ff2d4102ae0 2024-09-06T14:29:36.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:36.869+0000 7ff2db7f8640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff2d4112080 con 0x7ff2d4108e70 2024-09-06T14:29:36.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:36.869+0000 7ff2db7f8640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7ff2d4112380 con 0x7ff2d41084f0 2024-09-06T14:29:36.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:36.870+0000 7ff2d9594640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff2d4102ae0 0x7ff2d4198650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:29:36.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:36.870+0000 7ff2d8d93640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff2d41084f0 0x7ff2d4198b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:29:36.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:36.870+0000 7ff2d8d93640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff2d41084f0 0x7ff2d4198b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:42204/0 (socket says 172.21.15.142:42204) 2024-09-06T14:29:36.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:36.870+0000 7ff2d9594640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff2d4102ae0 0x7ff2d4198650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:58182/0 (socket says 172.21.15.142:58182) 2024-09-06T14:29:36.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:36.870+0000 7ff2d8d93640 1 -- 172.21.15.142:0/60443814 learned_addr learned my addr 172.21.15.142:0/60443814 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:29:36.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:36.870+0000 7ff2d9d95640 1 --2- 172.21.15.142:0/60443814 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff2d4108e70 0x7ff2d419df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:29:36.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:36.870+0000 7ff2d9594640 1 -- 172.21.15.142:0/60443814 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff2d41084f0 msgr2=0x7ff2d4198b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:29:36.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:36.870+0000 7ff2d9594640 1 --2- 172.21.15.142:0/60443814 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff2d41084f0 0x7ff2d4198b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:36.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:36.870+0000 7ff2d9594640 1 -- 172.21.15.142:0/60443814 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff2d4108e70 msgr2=0x7ff2d419df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:29:36.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:36.870+0000 7ff2d9594640 1 --2- 172.21.15.142:0/60443814 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff2d4108e70 0x7ff2d419df30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:36.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:36.870+0000 7ff2d9594640 1 -- 172.21.15.142:0/60443814 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2d419e700 con 0x7ff2d4102ae0 2024-09-06T14:29:36.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:36.870+0000 7ff2d8d93640 1 --2- 172.21.15.142:0/60443814 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff2d41084f0 0x7ff2d4198b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-06T14:29:36.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:36.870+0000 7ff2d9594640 1 --2- 172.21.15.142:0/60443814 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff2d4102ae0 0x7ff2d4198650 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ff2d000e9f0 tx=0x7ff2d000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:29:36.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:36.870+0000 7ff2ca7fc640 1 -- 172.21.15.142:0/60443814 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2d000cd80 con 0x7ff2d4102ae0 2024-09-06T14:29:36.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:36.870+0000 7ff2db7f8640 1 -- 172.21.15.142:0/60443814 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2d419e9f0 con 0x7ff2d4102ae0 2024-09-06T14:29:36.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:36.870+0000 7ff2ca7fc640 1 -- 172.21.15.142:0/60443814 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7ff2d00035f0 con 0x7ff2d4102ae0 2024-09-06T14:29:36.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:36.870+0000 7ff2ca7fc640 1 -- 172.21.15.142:0/60443814 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2d00044a0 con 0x7ff2d4102ae0 2024-09-06T14:29:36.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:36.870+0000 7ff2db7f8640 1 -- 172.21.15.142:0/60443814 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2d41a50b0 con 0x7ff2d4102ae0 2024-09-06T14:29:36.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:36.871+0000 7ff2db7f8640 1 -- 172.21.15.142:0/60443814 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2d4102ec0 con 0x7ff2d4102ae0 2024-09-06T14:29:36.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:36.872+0000 7ff2ca7fc640 1 -- 172.21.15.142:0/60443814 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 22) ==== 99844+0+0 (secure 0 0 0) 0x7ff2d0004000 con 0x7ff2d4102ae0 2024-09-06T14:29:36.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:36.872+0000 7ff2ca7fc640 1 --2- 172.21.15.142:0/60443814 >> v2:172.21.15.142:6800/2944545612 conn(0x7ff2c0077620 0x7ff2c0079ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:29:36.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:36.873+0000 7ff2d8d93640 1 --2- 172.21.15.142:0/60443814 >> v2:172.21.15.142:6800/2944545612 conn(0x7ff2c0077620 0x7ff2c0079ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:29:36.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:36.873+0000 7ff2ca7fc640 1 -- 172.21.15.142:0/60443814 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 6130+0+0 (secure 0 0 0) 0x7ff2d0099ce0 con 0x7ff2d4102ae0 2024-09-06T14:29:36.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:36.873+0000 7ff2d8d93640 1 --2- 172.21.15.142:0/60443814 >> v2:172.21.15.142:6800/2944545612 conn(0x7ff2c0077620 0x7ff2c0079ae0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7ff2d4101a90 tx=0x7ff2c400b410 comp rx=0 tx=0).ready entity=mgr.14601 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:29:36.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:36.876+0000 7ff2ca7fc640 1 -- 172.21.15.142:0/60443814 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7ff2d0062da0 con 0x7ff2d4102ae0 2024-09-06T14:29:37.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:29:36 smithi162 ceph-mon[33787]: pgmap v145: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:29:37.062 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:37.065+0000 7ff2db7f8640 1 -- 172.21.15.142:0/60443814 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7ff2d4199780 con 0x7ff2d4102ae0 2024-09-06T14:29:37.063 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:37.065+0000 7ff2ca7fc640 1 -- 172.21.15.142:0/60443814 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+212 (secure 0 0 0) 0x7ff2d006b250 con 0x7ff2d4102ae0 2024-09-06T14:29:37.063 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:29:37.063 INFO:teuthology.orchestra.run.smithi142.stdout:{"status":"HEALTH_WARN","checks":{"MGR_MODULE_DEPENDENCY":{"severity":"HEALTH_WARN","summary":{"message":"Module 'volumes' has failed dependency: No module named 'ceph.fs'","count":1},"muted":false}},"mutes":[]} 2024-09-06T14:29:37.065 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:37.067+0000 7ff2db7f8640 1 -- 172.21.15.142:0/60443814 >> v2:172.21.15.142:6800/2944545612 conn(0x7ff2c0077620 msgr2=0x7ff2c0079ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:29:37.065 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:37.067+0000 7ff2db7f8640 1 --2- 172.21.15.142:0/60443814 >> v2:172.21.15.142:6800/2944545612 conn(0x7ff2c0077620 0x7ff2c0079ae0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7ff2d4101a90 tx=0x7ff2c400b410 comp rx=0 tx=0).stop 2024-09-06T14:29:37.065 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:37.067+0000 7ff2db7f8640 1 -- 172.21.15.142:0/60443814 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff2d4102ae0 msgr2=0x7ff2d4198650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:29:37.065 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:37.067+0000 7ff2db7f8640 1 --2- 172.21.15.142:0/60443814 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff2d4102ae0 0x7ff2d4198650 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ff2d000e9f0 tx=0x7ff2d000eeb0 comp rx=0 tx=0).stop 2024-09-06T14:29:37.065 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:37.067+0000 7ff2db7f8640 1 -- 172.21.15.142:0/60443814 shutdown_connections 2024-09-06T14:29:37.065 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:37.067+0000 7ff2db7f8640 1 --2- 172.21.15.142:0/60443814 >> v2:172.21.15.142:6800/2944545612 conn(0x7ff2c0077620 0x7ff2c0079ae0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:37.066 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:37.067+0000 7ff2db7f8640 1 --2- 172.21.15.142:0/60443814 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff2d4108e70 0x7ff2d419df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:37.066 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:37.067+0000 7ff2db7f8640 1 --2- 172.21.15.142:0/60443814 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff2d41084f0 0x7ff2d4198b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:37.066 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:37.067+0000 7ff2db7f8640 1 --2- 172.21.15.142:0/60443814 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff2d4102ae0 0x7ff2d4198650 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:37.066 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:37.067+0000 7ff2db7f8640 1 -- 172.21.15.142:0/60443814 >> 172.21.15.142:0/60443814 conn(0x7ff2d40fe530 msgr2=0x7ff2d4100920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:29:37.066 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:37.067+0000 7ff2db7f8640 1 -- 172.21.15.142:0/60443814 shutdown_connections 2024-09-06T14:29:37.066 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:37.067+0000 7ff2db7f8640 1 -- 172.21.15.142:0/60443814 wait complete. 2024-09-06T14:29:37.548 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 06 14:29:37 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-iscsi-iscsi-a[59853]: debug there is no tcmu-runner data available 2024-09-06T14:29:37.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:29:37 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/60443814' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:29:37.959 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:29:37 smithi142 ceph-mon[37372]: from='client.? 172.21.15.142:0/60443814' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:29:38.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:29:37 smithi162 ceph-mon[33787]: from='client.? 172.21.15.142:0/60443814' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:29:38.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:29:38 smithi142 ceph-mon[29958]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-06T14:29:38.960 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:29:38 smithi142 ceph-mon[29958]: pgmap v146: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:29:38.960 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:29:38 smithi142 ceph-mon[37372]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-06T14:29:38.960 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:29:38 smithi142 ceph-mon[37372]: pgmap v146: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:29:39.048 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:29:38 smithi162 ceph-mon[33787]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-06T14:29:39.048 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:29:38 smithi162 ceph-mon[33787]: pgmap v146: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:29:40.734 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph health --format=json 2024-09-06T14:29:40.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:29:40 smithi142 ceph-mon[29958]: pgmap v147: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:29:40.959 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:29:40 smithi142 ceph-mon[37372]: pgmap v147: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:29:40.995 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:29:41.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:29:40 smithi162 ceph-mon[33787]: pgmap v147: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:29:41.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:41.481+0000 7fb692538640 1 -- 172.21.15.142:0/341966462 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb68c10ce70 msgr2=0x7fb68c10f2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:29:41.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:41.481+0000 7fb692538640 1 --2- 172.21.15.142:0/341966462 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb68c10ce70 0x7fb68c10f2a0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fb684009ca0 tx=0x7fb68402a8c0 comp rx=0 tx=0).stop 2024-09-06T14:29:41.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:41.481+0000 7fb692538640 1 -- 172.21.15.142:0/341966462 shutdown_connections 2024-09-06T14:29:41.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:41.481+0000 7fb692538640 1 --2- 172.21.15.142:0/341966462 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb68c10ce70 0x7fb68c10f2a0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:41.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:41.481+0000 7fb692538640 1 --2- 172.21.15.142:0/341966462 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb68c1042c0 0x7fb68c10c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:41.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:41.481+0000 7fb692538640 1 --2- 172.21.15.142:0/341966462 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb68c1038c0 0x7fb68c103ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:41.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:41.481+0000 7fb692538640 1 -- 172.21.15.142:0/341966462 >> 172.21.15.142:0/341966462 conn(0x7fb68c0fd430 msgr2=0x7fb68c0ff850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:29:41.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:41.481+0000 7fb692538640 1 -- 172.21.15.142:0/341966462 shutdown_connections 2024-09-06T14:29:41.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:41.481+0000 7fb692538640 1 -- 172.21.15.142:0/341966462 wait complete. 2024-09-06T14:29:41.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:41.482+0000 7fb692538640 1 Processor -- start 2024-09-06T14:29:41.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:41.482+0000 7fb692538640 1 -- start start 2024-09-06T14:29:41.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:41.482+0000 7fb692538640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb68c1038c0 0x7fb68c198780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:29:41.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:41.482+0000 7fb692538640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb68c1042c0 0x7fb68c198cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:29:41.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:41.482+0000 7fb692538640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb68c10ce70 0x7fb68c19e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:29:41.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:41.482+0000 7fb692538640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fb68c1121c0 con 0x7fb68c1038c0 2024-09-06T14:29:41.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:41.482+0000 7fb692538640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb68c112040 con 0x7fb68c10ce70 2024-09-06T14:29:41.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:41.482+0000 7fb692538640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7fb68c112340 con 0x7fb68c1042c0 2024-09-06T14:29:41.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:41.482+0000 7fb68bfff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb68c1038c0 0x7fb68c198780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:29:41.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:41.482+0000 7fb68bfff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb68c1038c0 0x7fb68c198780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:44170/0 (socket says 172.21.15.142:44170) 2024-09-06T14:29:41.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:41.482+0000 7fb68bfff640 1 -- 172.21.15.142:0/4126129937 learned_addr learned my addr 172.21.15.142:0/4126129937 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:29:41.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:41.482+0000 7fb68b7fe640 1 --2- 172.21.15.142:0/4126129937 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb68c1042c0 0x7fb68c198cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:29:41.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:41.482+0000 7fb690ad5640 1 --2- 172.21.15.142:0/4126129937 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb68c10ce70 0x7fb68c19e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:29:41.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:41.482+0000 7fb68bfff640 1 -- 172.21.15.142:0/4126129937 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb68c1042c0 msgr2=0x7fb68c198cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:29:41.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:41.482+0000 7fb68bfff640 1 --2- 172.21.15.142:0/4126129937 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb68c1042c0 0x7fb68c198cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:41.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:41.482+0000 7fb68bfff640 1 -- 172.21.15.142:0/4126129937 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb68c10ce70 msgr2=0x7fb68c19e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:29:41.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:41.482+0000 7fb68bfff640 1 --2- 172.21.15.142:0/4126129937 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb68c10ce70 0x7fb68c19e060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:41.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:41.482+0000 7fb68bfff640 1 -- 172.21.15.142:0/4126129937 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb68c19e830 con 0x7fb68c1038c0 2024-09-06T14:29:41.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:41.483+0000 7fb68bfff640 1 --2- 172.21.15.142:0/4126129937 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb68c1038c0 0x7fb68c198780 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fb67c00ea60 tx=0x7fb67c00ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:29:41.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:41.483+0000 7fb6897fa640 1 -- 172.21.15.142:0/4126129937 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb67c00cd20 con 0x7fb68c1038c0 2024-09-06T14:29:41.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:41.483+0000 7fb692538640 1 -- 172.21.15.142:0/4126129937 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb68c19eb20 con 0x7fb68c1038c0 2024-09-06T14:29:41.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:41.483+0000 7fb6897fa640 1 -- 172.21.15.142:0/4126129937 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7fb67c0035f0 con 0x7fb68c1038c0 2024-09-06T14:29:41.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:41.483+0000 7fb6897fa640 1 -- 172.21.15.142:0/4126129937 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb67c0044a0 con 0x7fb68c1038c0 2024-09-06T14:29:41.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:41.483+0000 7fb692538640 1 -- 172.21.15.142:0/4126129937 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb68c1a51e0 con 0x7fb68c1038c0 2024-09-06T14:29:41.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:41.484+0000 7fb692538640 1 -- 172.21.15.142:0/4126129937 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb68c103ca0 con 0x7fb68c1038c0 2024-09-06T14:29:41.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:41.484+0000 7fb6897fa640 1 -- 172.21.15.142:0/4126129937 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 22) ==== 99844+0+0 (secure 0 0 0) 0x7fb67c004000 con 0x7fb68c1038c0 2024-09-06T14:29:41.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:41.485+0000 7fb6897fa640 1 --2- 172.21.15.142:0/4126129937 >> v2:172.21.15.142:6800/2944545612 conn(0x7fb674077620 0x7fb674079ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:29:41.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:41.485+0000 7fb68b7fe640 1 --2- 172.21.15.142:0/4126129937 >> v2:172.21.15.142:6800/2944545612 conn(0x7fb674077620 0x7fb674079ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:29:41.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:41.485+0000 7fb6897fa640 1 -- 172.21.15.142:0/4126129937 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 6130+0+0 (secure 0 0 0) 0x7fb67c099d80 con 0x7fb68c1038c0 2024-09-06T14:29:41.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:41.485+0000 7fb68b7fe640 1 --2- 172.21.15.142:0/4126129937 >> v2:172.21.15.142:6800/2944545612 conn(0x7fb674077620 0x7fb674079ae0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fb68c100f90 tx=0x7fb680009110 comp rx=0 tx=0).ready entity=mgr.14601 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:29:41.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:41.489+0000 7fb6897fa640 1 -- 172.21.15.142:0/4126129937 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7fb67c062e40 con 0x7fb68c1038c0 2024-09-06T14:29:41.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:41.679+0000 7fb692538640 1 -- 172.21.15.142:0/4126129937 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fb68c107670 con 0x7fb68c1038c0 2024-09-06T14:29:41.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:41.679+0000 7fb6897fa640 1 -- 172.21.15.142:0/4126129937 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+212 (secure 0 0 0) 0x7fb67c083e20 con 0x7fb68c1038c0 2024-09-06T14:29:41.678 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:29:41.678 INFO:teuthology.orchestra.run.smithi142.stdout:{"status":"HEALTH_WARN","checks":{"MGR_MODULE_DEPENDENCY":{"severity":"HEALTH_WARN","summary":{"message":"Module 'volumes' has failed dependency: No module named 'ceph.fs'","count":1},"muted":false}},"mutes":[]} 2024-09-06T14:29:41.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:41.681+0000 7fb692538640 1 -- 172.21.15.142:0/4126129937 >> v2:172.21.15.142:6800/2944545612 conn(0x7fb674077620 msgr2=0x7fb674079ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:29:41.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:41.681+0000 7fb692538640 1 --2- 172.21.15.142:0/4126129937 >> v2:172.21.15.142:6800/2944545612 conn(0x7fb674077620 0x7fb674079ae0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fb68c100f90 tx=0x7fb680009110 comp rx=0 tx=0).stop 2024-09-06T14:29:41.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:41.681+0000 7fb692538640 1 -- 172.21.15.142:0/4126129937 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb68c1038c0 msgr2=0x7fb68c198780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:29:41.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:41.681+0000 7fb692538640 1 --2- 172.21.15.142:0/4126129937 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb68c1038c0 0x7fb68c198780 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fb67c00ea60 tx=0x7fb67c00ee60 comp rx=0 tx=0).stop 2024-09-06T14:29:41.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:41.682+0000 7fb692538640 1 -- 172.21.15.142:0/4126129937 shutdown_connections 2024-09-06T14:29:41.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:41.682+0000 7fb692538640 1 --2- 172.21.15.142:0/4126129937 >> v2:172.21.15.142:6800/2944545612 conn(0x7fb674077620 0x7fb674079ae0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:41.680 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:41.682+0000 7fb692538640 1 --2- 172.21.15.142:0/4126129937 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb68c10ce70 0x7fb68c19e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:41.680 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:41.682+0000 7fb692538640 1 --2- 172.21.15.142:0/4126129937 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb68c1042c0 0x7fb68c198cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:41.680 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:41.682+0000 7fb692538640 1 --2- 172.21.15.142:0/4126129937 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb68c1038c0 0x7fb68c198780 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:41.680 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:41.682+0000 7fb692538640 1 -- 172.21.15.142:0/4126129937 >> 172.21.15.142:0/4126129937 conn(0x7fb68c0fd430 msgr2=0x7fb68c104840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:29:41.680 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:41.682+0000 7fb692538640 1 -- 172.21.15.142:0/4126129937 shutdown_connections 2024-09-06T14:29:41.680 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:41.682+0000 7fb692538640 1 -- 172.21.15.142:0/4126129937 wait complete. 2024-09-06T14:29:41.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:29:41 smithi142 ceph-mon[29958]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-06T14:29:41.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:29:41 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/4126129937' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:29:41.960 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:29:41 smithi142 ceph-mon[37372]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-06T14:29:41.960 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:29:41 smithi142 ceph-mon[37372]: from='client.? 172.21.15.142:0/4126129937' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:29:42.048 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:29:41 smithi162 ceph-mon[33787]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-06T14:29:42.048 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:29:41 smithi162 ceph-mon[33787]: from='client.? 172.21.15.142:0/4126129937' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:29:42.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:29:42 smithi142 ceph-mon[29958]: pgmap v148: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:29:42.959 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:29:42 smithi142 ceph-mon[37372]: pgmap v148: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:29:43.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:29:42 smithi162 ceph-mon[33787]: pgmap v148: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:29:44.709 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:29:44 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: ::ffff:172.21.15.162 - - [06/Sep/2024:14:29:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-06T14:29:45.048 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:29:44 smithi162 ceph-mon[33787]: pgmap v149: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:29:45.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:29:44 smithi142 ceph-mon[29958]: pgmap v149: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:29:45.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:29:44 smithi142 ceph-mon[37372]: pgmap v149: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:29:45.381 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph health --format=json 2024-09-06T14:29:45.642 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:29:46.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:46.129+0000 7f0675e69640 1 -- 172.21.15.142:0/3719905755 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0670107d70 msgr2=0x7f067010e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:29:46.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:46.129+0000 7f0675e69640 1 --2- 172.21.15.142:0/3719905755 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0670107d70 0x7f067010e570 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f0668009ca0 tx=0x7f066802a8c0 comp rx=0 tx=0).stop 2024-09-06T14:29:46.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:46.129+0000 7f0675e69640 1 -- 172.21.15.142:0/3719905755 shutdown_connections 2024-09-06T14:29:46.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:46.129+0000 7f0675e69640 1 --2- 172.21.15.142:0/3719905755 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0670107d70 0x7f067010e570 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:46.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:46.129+0000 7f0675e69640 1 --2- 172.21.15.142:0/3719905755 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f06701073f0 0x7f0670107830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:46.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:46.129+0000 7f0675e69640 1 --2- 172.21.15.142:0/3719905755 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f06701019e0 0x7f0670101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:46.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:46.129+0000 7f0675e69640 1 -- 172.21.15.142:0/3719905755 >> 172.21.15.142:0/3719905755 conn(0x7f06700fd450 msgr2=0x7f06700ff870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:29:46.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:46.129+0000 7f0675e69640 1 -- 172.21.15.142:0/3719905755 shutdown_connections 2024-09-06T14:29:46.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:46.129+0000 7f0675e69640 1 -- 172.21.15.142:0/3719905755 wait complete. 2024-09-06T14:29:46.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:46.129+0000 7f0675e69640 1 Processor -- start 2024-09-06T14:29:46.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:46.130+0000 7f0675e69640 1 -- start start 2024-09-06T14:29:46.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:46.130+0000 7f0675e69640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f06701019e0 0x7f067010f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:29:46.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:46.130+0000 7f0675e69640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f06701073f0 0x7f067010f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:29:46.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:46.130+0000 7f0675e69640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0670107d70 0x7f067010fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:29:46.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:46.130+0000 7f0675e69640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f0670112240 con 0x7f0670107d70 2024-09-06T14:29:46.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:46.130+0000 7f0675e69640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f06701120c0 con 0x7f06701073f0 2024-09-06T14:29:46.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:46.130+0000 7f0675e69640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7f06701123c0 con 0x7f06701019e0 2024-09-06T14:29:46.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:46.130+0000 7f066f7fe640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f06701019e0 0x7f067010f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:29:46.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:46.130+0000 7f066ffff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0670107d70 0x7f067010fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:29:46.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:46.130+0000 7f066f7fe640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f06701019e0 0x7f067010f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:59928/0 (socket says 172.21.15.142:59928) 2024-09-06T14:29:46.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:46.130+0000 7f066f7fe640 1 -- 172.21.15.142:0/542660699 learned_addr learned my addr 172.21.15.142:0/542660699 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:29:46.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:46.130+0000 7f066effd640 1 --2- 172.21.15.142:0/542660699 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f06701073f0 0x7f067010f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:29:46.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:46.130+0000 7f066f7fe640 1 -- 172.21.15.142:0/542660699 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f06701073f0 msgr2=0x7f067010f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:29:46.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:46.130+0000 7f066f7fe640 1 --2- 172.21.15.142:0/542660699 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f06701073f0 0x7f067010f5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:46.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:46.130+0000 7f066f7fe640 1 -- 172.21.15.142:0/542660699 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0670107d70 msgr2=0x7f067010fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:29:46.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:46.130+0000 7f066f7fe640 1 --2- 172.21.15.142:0/542660699 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0670107d70 0x7f067010fb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:46.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:46.130+0000 7f066f7fe640 1 -- 172.21.15.142:0/542660699 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06701a6ad0 con 0x7f06701019e0 2024-09-06T14:29:46.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:46.130+0000 7f066ffff640 1 --2- 172.21.15.142:0/542660699 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0670107d70 0x7f067010fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-06T14:29:46.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:46.131+0000 7f066f7fe640 1 --2- 172.21.15.142:0/542660699 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f06701019e0 0x7f067010f080 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f066000b4a0 tx=0x7f066000b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:29:46.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:46.131+0000 7f066cff9640 1 -- 172.21.15.142:0/542660699 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0660014070 con 0x7f06701019e0 2024-09-06T14:29:46.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:46.131+0000 7f066cff9640 1 -- 172.21.15.142:0/542660699 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7f0660010a90 con 0x7f06701019e0 2024-09-06T14:29:46.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:46.131+0000 7f0675e69640 1 -- 172.21.15.142:0/542660699 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06701a6dc0 con 0x7f06701019e0 2024-09-06T14:29:46.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:46.131+0000 7f066cff9640 1 -- 172.21.15.142:0/542660699 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0660017b80 con 0x7f06701019e0 2024-09-06T14:29:46.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:46.131+0000 7f0675e69640 1 -- 172.21.15.142:0/542660699 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f06701a7300 con 0x7f06701019e0 2024-09-06T14:29:46.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:46.132+0000 7f0675e69640 1 -- 172.21.15.142:0/542660699 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0670101dc0 con 0x7f06701019e0 2024-09-06T14:29:46.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:46.134+0000 7f066cff9640 1 -- 172.21.15.142:0/542660699 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 22) ==== 99844+0+0 (secure 0 0 0) 0x7f0660017070 con 0x7f06701019e0 2024-09-06T14:29:46.135 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:46.134+0000 7f066cff9640 1 --2- 172.21.15.142:0/542660699 >> v2:172.21.15.142:6800/2944545612 conn(0x7f0658077620 0x7f0658079ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:29:46.135 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:46.134+0000 7f066cff9640 1 -- 172.21.15.142:0/542660699 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(64..64 src has 1..64) ==== 6130+0+0 (secure 0 0 0) 0x7f0660098e00 con 0x7f06701019e0 2024-09-06T14:29:46.135 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:46.134+0000 7f066effd640 1 --2- 172.21.15.142:0/542660699 >> v2:172.21.15.142:6800/2944545612 conn(0x7f0658077620 0x7f0658079ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:29:46.135 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:46.135+0000 7f066effd640 1 --2- 172.21.15.142:0/542660699 >> v2:172.21.15.142:6800/2944545612 conn(0x7f0658077620 0x7f0658079ae0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f0670110b70 tx=0x7f0664005f90 comp rx=0 tx=0).ready entity=mgr.14601 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:29:46.135 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:46.137+0000 7f066cff9640 1 -- 172.21.15.142:0/542660699 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7f0660066190 con 0x7f06701019e0 2024-09-06T14:29:46.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:46.325+0000 7f0675e69640 1 -- 172.21.15.142:0/542660699 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f0670110380 con 0x7f06701019e0 2024-09-06T14:29:46.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:46.326+0000 7f066cff9640 1 -- 172.21.15.142:0/542660699 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+212 (secure 0 0 0) 0x7f0670110380 con 0x7f06701019e0 2024-09-06T14:29:46.324 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:29:46.324 INFO:teuthology.orchestra.run.smithi142.stdout:{"status":"HEALTH_WARN","checks":{"MGR_MODULE_DEPENDENCY":{"severity":"HEALTH_WARN","summary":{"message":"Module 'volumes' has failed dependency: No module named 'ceph.fs'","count":1},"muted":false}},"mutes":[]} 2024-09-06T14:29:46.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:46.329+0000 7f0675e69640 1 -- 172.21.15.142:0/542660699 >> v2:172.21.15.142:6800/2944545612 conn(0x7f0658077620 msgr2=0x7f0658079ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:29:46.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:46.329+0000 7f0675e69640 1 --2- 172.21.15.142:0/542660699 >> v2:172.21.15.142:6800/2944545612 conn(0x7f0658077620 0x7f0658079ae0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f0670110b70 tx=0x7f0664005f90 comp rx=0 tx=0).stop 2024-09-06T14:29:46.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:46.329+0000 7f0675e69640 1 -- 172.21.15.142:0/542660699 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f06701019e0 msgr2=0x7f067010f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:29:46.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:46.329+0000 7f0675e69640 1 --2- 172.21.15.142:0/542660699 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f06701019e0 0x7f067010f080 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f066000b4a0 tx=0x7f066000b960 comp rx=0 tx=0).stop 2024-09-06T14:29:46.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:46.329+0000 7f0675e69640 1 -- 172.21.15.142:0/542660699 shutdown_connections 2024-09-06T14:29:46.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:46.329+0000 7f0675e69640 1 --2- 172.21.15.142:0/542660699 >> v2:172.21.15.142:6800/2944545612 conn(0x7f0658077620 0x7f0658079ae0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:46.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:46.329+0000 7f0675e69640 1 --2- 172.21.15.142:0/542660699 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0670107d70 0x7f067010fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:46.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:46.329+0000 7f0675e69640 1 --2- 172.21.15.142:0/542660699 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f06701073f0 0x7f067010f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:46.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:46.329+0000 7f0675e69640 1 --2- 172.21.15.142:0/542660699 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f06701019e0 0x7f067010f080 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:46.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:46.329+0000 7f0675e69640 1 -- 172.21.15.142:0/542660699 >> 172.21.15.142:0/542660699 conn(0x7f06700fd450 msgr2=0x7f0670104a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:29:46.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:46.329+0000 7f0675e69640 1 -- 172.21.15.142:0/542660699 shutdown_connections 2024-09-06T14:29:46.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:46.329+0000 7f0675e69640 1 -- 172.21.15.142:0/542660699 wait complete. 2024-09-06T14:29:47.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:29:46 smithi162 ceph-mon[33787]: pgmap v150: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:29:47.048 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:29:46 smithi162 ceph-mon[33787]: from='client.? 172.21.15.142:0/542660699' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:29:47.091 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:29:46 smithi142 ceph-mon[29958]: pgmap v150: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:29:47.091 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:29:46 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/542660699' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:29:47.091 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:29:46 smithi142 ceph-mon[37372]: pgmap v150: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:29:47.091 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:29:46 smithi142 ceph-mon[37372]: from='client.? 172.21.15.142:0/542660699' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:29:47.547 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 06 14:29:47 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-iscsi-iscsi-a[59853]: debug there is no tcmu-runner data available 2024-09-06T14:29:49.048 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:29:48 smithi162 ceph-mon[33787]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-06T14:29:49.048 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:29:48 smithi162 ceph-mon[33787]: pgmap v151: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:29:49.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:29:48 smithi142 ceph-mon[29958]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-06T14:29:49.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:29:48 smithi142 ceph-mon[29958]: pgmap v151: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:29:49.210 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:29:48 smithi142 ceph-mon[37372]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-06T14:29:49.210 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:29:48 smithi142 ceph-mon[37372]: pgmap v151: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:29:50.121 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph health --format=json 2024-09-06T14:29:50.381 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:29:50.779 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:29:50 smithi142 ceph-mon[29958]: pgmap v152: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:29:50.780 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:29:50 smithi142 ceph-mon[37372]: pgmap v152: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:29:50.880 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:50.860+0000 7f5c4851c640 1 -- 172.21.15.142:0/542487879 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5c401019c0 msgr2=0x7f5c40101da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:29:50.880 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:50.860+0000 7f5c4851c640 1 --2- 172.21.15.142:0/542487879 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5c401019c0 0x7f5c40101da0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f5c3c009860 tx=0x7f5c3c02a340 comp rx=0 tx=0).stop 2024-09-06T14:29:50.880 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:50.860+0000 7f5c4851c640 1 -- 172.21.15.142:0/542487879 shutdown_connections 2024-09-06T14:29:50.880 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:50.860+0000 7f5c4851c640 1 --2- 172.21.15.142:0/542487879 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5c40107d50 0x7f5c4010e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:50.880 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:50.860+0000 7f5c4851c640 1 --2- 172.21.15.142:0/542487879 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5c401073d0 0x7f5c40107810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:50.881 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:50.860+0000 7f5c4851c640 1 --2- 172.21.15.142:0/542487879 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5c401019c0 0x7f5c40101da0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:50.881 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:50.860+0000 7f5c4851c640 1 -- 172.21.15.142:0/542487879 >> 172.21.15.142:0/542487879 conn(0x7f5c400fd430 msgr2=0x7f5c400ff850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:29:50.881 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:50.860+0000 7f5c4851c640 1 -- 172.21.15.142:0/542487879 shutdown_connections 2024-09-06T14:29:50.881 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:50.860+0000 7f5c4851c640 1 -- 172.21.15.142:0/542487879 wait complete. 2024-09-06T14:29:50.881 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:50.861+0000 7f5c4851c640 1 Processor -- start 2024-09-06T14:29:50.881 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:50.861+0000 7f5c4851c640 1 -- start start 2024-09-06T14:29:50.881 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:50.861+0000 7f5c4851c640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5c401019c0 0x7f5c40198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:29:50.881 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:50.861+0000 7f5c4851c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5c401073d0 0x7f5c40198c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:29:50.881 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:50.861+0000 7f5c4851c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5c40107d50 0x7f5c4019dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:29:50.881 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:50.861+0000 7f5c4851c640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f5c40111120 con 0x7f5c40107d50 2024-09-06T14:29:50.881 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:50.861+0000 7f5c4851c640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5c40110fa0 con 0x7f5c401073d0 2024-09-06T14:29:50.881 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:50.861+0000 7f5c4851c640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7f5c401112a0 con 0x7f5c401019c0 2024-09-06T14:29:50.881 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:50.861+0000 7f5c462b8640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5c401019c0 0x7f5c40198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:29:50.881 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:50.861+0000 7f5c462b8640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5c401019c0 0x7f5c40198700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:55586/0 (socket says 172.21.15.142:55586) 2024-09-06T14:29:50.882 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:50.861+0000 7f5c462b8640 1 -- 172.21.15.142:0/3958005388 learned_addr learned my addr 172.21.15.142:0/3958005388 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:29:50.882 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:50.861+0000 7f5c46ab9640 1 --2- 172.21.15.142:0/3958005388 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5c40107d50 0x7f5c4019dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:29:50.882 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:50.861+0000 7f5c45ab7640 1 --2- 172.21.15.142:0/3958005388 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5c401073d0 0x7f5c40198c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:29:50.882 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:50.861+0000 7f5c462b8640 1 -- 172.21.15.142:0/3958005388 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5c401073d0 msgr2=0x7f5c40198c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:29:50.882 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:50.861+0000 7f5c462b8640 1 --2- 172.21.15.142:0/3958005388 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5c401073d0 0x7f5c40198c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:50.882 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:50.861+0000 7f5c462b8640 1 -- 172.21.15.142:0/3958005388 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5c40107d50 msgr2=0x7f5c4019dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:29:50.882 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:50.861+0000 7f5c462b8640 1 --2- 172.21.15.142:0/3958005388 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5c40107d50 0x7f5c4019dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:50.882 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:50.861+0000 7f5c462b8640 1 -- 172.21.15.142:0/3958005388 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c4019e7b0 con 0x7f5c401019c0 2024-09-06T14:29:50.882 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:50.862+0000 7f5c462b8640 1 --2- 172.21.15.142:0/3958005388 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5c401019c0 0x7f5c40198700 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f5c3c009980 tx=0x7f5c3c033d30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:29:50.882 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:50.862+0000 7f5c377fe640 1 -- 172.21.15.142:0/3958005388 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c3c03f020 con 0x7f5c401019c0 2024-09-06T14:29:50.882 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:50.862+0000 7f5c377fe640 1 -- 172.21.15.142:0/3958005388 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7f5c3c0044c0 con 0x7f5c401019c0 2024-09-06T14:29:50.882 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:50.862+0000 7f5c4851c640 1 -- 172.21.15.142:0/3958005388 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c4019ea40 con 0x7f5c401019c0 2024-09-06T14:29:50.882 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:50.862+0000 7f5c377fe640 1 -- 172.21.15.142:0/3958005388 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c3c037040 con 0x7f5c401019c0 2024-09-06T14:29:50.882 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:50.862+0000 7f5c4851c640 1 -- 172.21.15.142:0/3958005388 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c401a5160 con 0x7f5c401019c0 2024-09-06T14:29:50.883 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:50.863+0000 7f5c4851c640 1 -- 172.21.15.142:0/3958005388 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c40101da0 con 0x7f5c401019c0 2024-09-06T14:29:50.883 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:50.863+0000 7f5c377fe640 1 -- 172.21.15.142:0/3958005388 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 22) ==== 99844+0+0 (secure 0 0 0) 0x7f5c3c042020 con 0x7f5c401019c0 2024-09-06T14:29:50.883 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:50.864+0000 7f5c377fe640 1 --2- 172.21.15.142:0/3958005388 >> v2:172.21.15.142:6800/2944545612 conn(0x7f5c2c077620 0x7f5c2c079ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:29:50.883 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:50.864+0000 7f5c45ab7640 1 --2- 172.21.15.142:0/3958005388 >> v2:172.21.15.142:6800/2944545612 conn(0x7f5c2c077620 0x7f5c2c079ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:29:50.883 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:50.864+0000 7f5c377fe640 1 -- 172.21.15.142:0/3958005388 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(64..64 src has 1..64) ==== 6130+0+0 (secure 0 0 0) 0x7f5c3c0b9980 con 0x7f5c401019c0 2024-09-06T14:29:50.883 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:50.864+0000 7f5c45ab7640 1 --2- 172.21.15.142:0/3958005388 >> v2:172.21.15.142:6800/2944545612 conn(0x7f5c2c077620 0x7f5c2c079ae0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f5c400fe750 tx=0x7f5c3000a4c0 comp rx=0 tx=0).ready entity=mgr.14601 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:29:50.883 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:50.868+0000 7f5c377fe640 1 -- 172.21.15.142:0/3958005388 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7f5c3c049020 con 0x7f5c401019c0 2024-09-06T14:29:51.048 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:29:50 smithi162 ceph-mon[33787]: pgmap v152: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:29:51.053 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:51.055+0000 7f5c4851c640 1 -- 172.21.15.142:0/3958005388 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f5c401a5390 con 0x7f5c401019c0 2024-09-06T14:29:51.053 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:51.056+0000 7f5c377fe640 1 -- 172.21.15.142:0/3958005388 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+212 (secure 0 0 0) 0x7f5c3c082ac0 con 0x7f5c401019c0 2024-09-06T14:29:51.053 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:29:51.053 INFO:teuthology.orchestra.run.smithi142.stdout:{"status":"HEALTH_WARN","checks":{"MGR_MODULE_DEPENDENCY":{"severity":"HEALTH_WARN","summary":{"message":"Module 'volumes' has failed dependency: No module named 'ceph.fs'","count":1},"muted":false}},"mutes":[]} 2024-09-06T14:29:51.055 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:51.058+0000 7f5c4851c640 1 -- 172.21.15.142:0/3958005388 >> v2:172.21.15.142:6800/2944545612 conn(0x7f5c2c077620 msgr2=0x7f5c2c079ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:29:51.055 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:51.058+0000 7f5c4851c640 1 --2- 172.21.15.142:0/3958005388 >> v2:172.21.15.142:6800/2944545612 conn(0x7f5c2c077620 0x7f5c2c079ae0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f5c400fe750 tx=0x7f5c3000a4c0 comp rx=0 tx=0).stop 2024-09-06T14:29:51.055 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:51.058+0000 7f5c4851c640 1 -- 172.21.15.142:0/3958005388 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5c401019c0 msgr2=0x7f5c40198700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:29:51.055 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:51.058+0000 7f5c4851c640 1 --2- 172.21.15.142:0/3958005388 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5c401019c0 0x7f5c40198700 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f5c3c009980 tx=0x7f5c3c033d30 comp rx=0 tx=0).stop 2024-09-06T14:29:51.055 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:51.058+0000 7f5c4851c640 1 -- 172.21.15.142:0/3958005388 shutdown_connections 2024-09-06T14:29:51.055 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:51.058+0000 7f5c4851c640 1 --2- 172.21.15.142:0/3958005388 >> v2:172.21.15.142:6800/2944545612 conn(0x7f5c2c077620 0x7f5c2c079ae0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:51.056 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:51.058+0000 7f5c4851c640 1 --2- 172.21.15.142:0/3958005388 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5c40107d50 0x7f5c4019dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:51.056 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:51.058+0000 7f5c4851c640 1 --2- 172.21.15.142:0/3958005388 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5c401073d0 0x7f5c40198c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:51.056 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:51.058+0000 7f5c4851c640 1 --2- 172.21.15.142:0/3958005388 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5c401019c0 0x7f5c40198700 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:51.056 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:51.058+0000 7f5c4851c640 1 -- 172.21.15.142:0/3958005388 >> 172.21.15.142:0/3958005388 conn(0x7f5c400fd430 msgr2=0x7f5c4006a8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:29:51.056 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:51.058+0000 7f5c4851c640 1 -- 172.21.15.142:0/3958005388 shutdown_connections 2024-09-06T14:29:51.056 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:51.058+0000 7f5c4851c640 1 -- 172.21.15.142:0/3958005388 wait complete. 2024-09-06T14:29:51.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:29:51 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/3958005388' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:29:51.959 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:29:51 smithi142 ceph-mon[37372]: from='client.? 172.21.15.142:0/3958005388' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:29:52.048 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:29:51 smithi162 ceph-mon[33787]: from='client.? 172.21.15.142:0/3958005388' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:29:53.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:29:52 smithi162 ceph-mon[33787]: pgmap v153: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:29:53.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:29:52 smithi142 ceph-mon[29958]: pgmap v153: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:29:53.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:29:52 smithi142 ceph-mon[37372]: pgmap v153: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:29:54.709 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:29:54 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: ::ffff:172.21.15.162 - - [06/Sep/2024:14:29:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-06T14:29:54.769 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph health --format=json 2024-09-06T14:29:54.994 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:29:54 smithi142 ceph-mon[29958]: pgmap v154: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:29:54.994 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:29:54 smithi142 ceph-mon[37372]: pgmap v154: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:29:55.040 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:29:55.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:29:54 smithi162 ceph-mon[33787]: pgmap v154: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:29:55.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:55.524+0000 7f5661964640 1 -- 172.21.15.142:0/3018868778 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f565c107d90 msgr2=0x7f565c10e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:29:55.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:55.524+0000 7f5661964640 1 --2- 172.21.15.142:0/3018868778 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f565c107d90 0x7f565c10e590 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f5654009be0 tx=0x7f565402a6c0 comp rx=0 tx=0).stop 2024-09-06T14:29:55.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:55.524+0000 7f5661964640 1 -- 172.21.15.142:0/3018868778 shutdown_connections 2024-09-06T14:29:55.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:55.524+0000 7f5661964640 1 --2- 172.21.15.142:0/3018868778 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f565c107d90 0x7f565c10e590 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:55.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:55.524+0000 7f5661964640 1 --2- 172.21.15.142:0/3018868778 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f565c107410 0x7f565c107850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:55.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:55.524+0000 7f5661964640 1 --2- 172.21.15.142:0/3018868778 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f565c101a00 0x7f565c101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:55.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:55.524+0000 7f5661964640 1 -- 172.21.15.142:0/3018868778 >> 172.21.15.142:0/3018868778 conn(0x7f565c0fd470 msgr2=0x7f565c0ff890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:29:55.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:55.524+0000 7f5661964640 1 -- 172.21.15.142:0/3018868778 shutdown_connections 2024-09-06T14:29:55.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:55.524+0000 7f5661964640 1 -- 172.21.15.142:0/3018868778 wait complete. 2024-09-06T14:29:55.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:55.525+0000 7f5661964640 1 Processor -- start 2024-09-06T14:29:55.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:55.525+0000 7f5661964640 1 -- start start 2024-09-06T14:29:55.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:55.525+0000 7f5661964640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f565c101a00 0x7f565c198690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:29:55.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:55.525+0000 7f5661964640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f565c107410 0x7f565c198bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:29:55.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:55.525+0000 7f5661964640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f565c107d90 0x7f565c19df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:29:55.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:55.525+0000 7f5661964640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f565c111120 con 0x7f565c107410 2024-09-06T14:29:55.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:55.525+0000 7f5661964640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f565c110fa0 con 0x7f565c107d90 2024-09-06T14:29:55.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:55.525+0000 7f5661964640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7f565c1112a0 con 0x7f565c101a00 2024-09-06T14:29:55.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:55.525+0000 7f565a7fc640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f565c107410 0x7f565c198bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:29:55.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:55.525+0000 7f565affd640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f565c101a00 0x7f565c198690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:29:55.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:55.525+0000 7f565affd640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f565c101a00 0x7f565c198690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:55596/0 (socket says 172.21.15.142:55596) 2024-09-06T14:29:55.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:55.525+0000 7f565a7fc640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f565c107410 0x7f565c198bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:53416/0 (socket says 172.21.15.142:53416) 2024-09-06T14:29:55.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:55.525+0000 7f565affd640 1 -- 172.21.15.142:0/807408719 learned_addr learned my addr 172.21.15.142:0/807408719 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:29:55.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:55.526+0000 7f565b7fe640 1 --2- 172.21.15.142:0/807408719 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f565c107d90 0x7f565c19df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:29:55.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:55.526+0000 7f565affd640 1 -- 172.21.15.142:0/807408719 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f565c107d90 msgr2=0x7f565c19df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:29:55.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:55.526+0000 7f565affd640 1 --2- 172.21.15.142:0/807408719 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f565c107d90 0x7f565c19df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:55.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:55.526+0000 7f565affd640 1 -- 172.21.15.142:0/807408719 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f565c107410 msgr2=0x7f565c198bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:29:55.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:55.526+0000 7f565affd640 1 --2- 172.21.15.142:0/807408719 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f565c107410 0x7f565c198bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:55.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:55.526+0000 7f565affd640 1 -- 172.21.15.142:0/807408719 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f565c19e740 con 0x7f565c101a00 2024-09-06T14:29:55.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:55.526+0000 7f565b7fe640 1 --2- 172.21.15.142:0/807408719 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f565c107d90 0x7f565c19df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-06T14:29:55.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:55.526+0000 7f565a7fc640 1 --2- 172.21.15.142:0/807408719 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f565c107410 0x7f565c198bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-06T14:29:55.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:55.526+0000 7f565affd640 1 --2- 172.21.15.142:0/807408719 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f565c101a00 0x7f565c198690 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f564c00e9c0 tx=0x7f564c00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:29:55.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:55.526+0000 7f5660962640 1 -- 172.21.15.142:0/807408719 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f564c00cde0 con 0x7f565c101a00 2024-09-06T14:29:55.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:55.526+0000 7f5660962640 1 -- 172.21.15.142:0/807408719 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7f564c0035f0 con 0x7f565c101a00 2024-09-06T14:29:55.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:55.526+0000 7f5661964640 1 -- 172.21.15.142:0/807408719 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f565c19ea30 con 0x7f565c101a00 2024-09-06T14:29:55.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:55.526+0000 7f5660962640 1 -- 172.21.15.142:0/807408719 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f564c0044a0 con 0x7f565c101a00 2024-09-06T14:29:55.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:55.526+0000 7f5661964640 1 -- 172.21.15.142:0/807408719 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f565c1a52c0 con 0x7f565c101a00 2024-09-06T14:29:55.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:55.527+0000 7f5661964640 1 -- 172.21.15.142:0/807408719 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f565c101de0 con 0x7f565c101a00 2024-09-06T14:29:55.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:55.528+0000 7f5660962640 1 -- 172.21.15.142:0/807408719 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 22) ==== 99844+0+0 (secure 0 0 0) 0x7f564c004000 con 0x7f565c101a00 2024-09-06T14:29:55.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:55.529+0000 7f5660962640 1 --2- 172.21.15.142:0/807408719 >> v2:172.21.15.142:6800/2944545612 conn(0x7f5644077620 0x7f5644079ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:29:55.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:55.529+0000 7f565a7fc640 1 --2- 172.21.15.142:0/807408719 >> v2:172.21.15.142:6800/2944545612 conn(0x7f5644077620 0x7f5644079ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:29:55.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:55.529+0000 7f5660962640 1 -- 172.21.15.142:0/807408719 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(64..64 src has 1..64) ==== 6130+0+0 (secure 0 0 0) 0x7f564c099c90 con 0x7f565c101a00 2024-09-06T14:29:55.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:55.529+0000 7f565a7fc640 1 --2- 172.21.15.142:0/807408719 >> v2:172.21.15.142:6800/2944545612 conn(0x7f5644077620 0x7f5644079ae0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f565c100ea0 tx=0x7f5650005f90 comp rx=0 tx=0).ready entity=mgr.14601 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:29:55.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:55.533+0000 7f5660962640 1 -- 172.21.15.142:0/807408719 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7f564c09f020 con 0x7f565c101a00 2024-09-06T14:29:55.728 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:55.730+0000 7f5661964640 1 -- 172.21.15.142:0/807408719 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f565c10ab80 con 0x7f565c101a00 2024-09-06T14:29:55.729 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:55.730+0000 7f5660962640 1 -- 172.21.15.142:0/807408719 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+212 (secure 0 0 0) 0x7f564c062dd0 con 0x7f565c101a00 2024-09-06T14:29:55.729 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:29:55.729 INFO:teuthology.orchestra.run.smithi142.stdout:{"status":"HEALTH_WARN","checks":{"MGR_MODULE_DEPENDENCY":{"severity":"HEALTH_WARN","summary":{"message":"Module 'volumes' has failed dependency: No module named 'ceph.fs'","count":1},"muted":false}},"mutes":[]} 2024-09-06T14:29:55.730 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:55.733+0000 7f5661964640 1 -- 172.21.15.142:0/807408719 >> v2:172.21.15.142:6800/2944545612 conn(0x7f5644077620 msgr2=0x7f5644079ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:29:55.731 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:55.733+0000 7f5661964640 1 --2- 172.21.15.142:0/807408719 >> v2:172.21.15.142:6800/2944545612 conn(0x7f5644077620 0x7f5644079ae0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f565c100ea0 tx=0x7f5650005f90 comp rx=0 tx=0).stop 2024-09-06T14:29:55.731 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:55.733+0000 7f5661964640 1 -- 172.21.15.142:0/807408719 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f565c101a00 msgr2=0x7f565c198690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:29:55.731 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:55.733+0000 7f5661964640 1 --2- 172.21.15.142:0/807408719 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f565c101a00 0x7f565c198690 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f564c00e9c0 tx=0x7f564c00ee80 comp rx=0 tx=0).stop 2024-09-06T14:29:55.731 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:55.733+0000 7f5661964640 1 -- 172.21.15.142:0/807408719 shutdown_connections 2024-09-06T14:29:55.731 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:55.733+0000 7f5661964640 1 --2- 172.21.15.142:0/807408719 >> v2:172.21.15.142:6800/2944545612 conn(0x7f5644077620 0x7f5644079ae0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:55.731 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:55.733+0000 7f5661964640 1 --2- 172.21.15.142:0/807408719 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f565c107d90 0x7f565c19df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:55.731 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:55.733+0000 7f5661964640 1 --2- 172.21.15.142:0/807408719 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f565c107410 0x7f565c198bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:55.731 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:55.733+0000 7f5661964640 1 --2- 172.21.15.142:0/807408719 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f565c101a00 0x7f565c198690 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:29:55.731 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:55.733+0000 7f5661964640 1 -- 172.21.15.142:0/807408719 >> 172.21.15.142:0/807408719 conn(0x7f565c0fd470 msgr2=0x7f565c06b440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:29:55.731 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:55.733+0000 7f5661964640 1 -- 172.21.15.142:0/807408719 shutdown_connections 2024-09-06T14:29:55.731 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:29:55.733+0000 7f5661964640 1 -- 172.21.15.142:0/807408719 wait complete. 2024-09-06T14:29:55.990 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:29:55 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/807408719' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:29:55.990 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:29:55 smithi142 ceph-mon[37372]: from='client.? 172.21.15.142:0/807408719' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:29:56.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:29:55 smithi162 ceph-mon[33787]: from='client.? 172.21.15.142:0/807408719' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:29:56.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:29:56 smithi142 ceph-mon[29958]: pgmap v155: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:29:56.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:29:56 smithi142 ceph-mon[29958]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-06T14:29:56.960 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:29:56 smithi142 ceph-mon[37372]: pgmap v155: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:29:56.960 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:29:56 smithi142 ceph-mon[37372]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-06T14:29:57.048 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:29:56 smithi162 ceph-mon[33787]: pgmap v155: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:29:57.048 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:29:56 smithi162 ceph-mon[33787]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-06T14:29:57.548 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 06 14:29:57 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-iscsi-iscsi-a[59853]: debug there is no tcmu-runner data available 2024-09-06T14:29:59.048 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:29:58 smithi162 ceph-mon[33787]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-06T14:29:59.048 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:29:58 smithi162 ceph-mon[33787]: pgmap v156: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:29:59.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:29:58 smithi142 ceph-mon[37372]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-06T14:29:59.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:29:58 smithi142 ceph-mon[37372]: pgmap v156: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:29:59.210 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:29:58 smithi142 ceph-mon[29958]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-06T14:29:59.210 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:29:58 smithi142 ceph-mon[29958]: pgmap v156: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:29:59.535 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph health --format=json 2024-09-06T14:29:59.795 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:30:00.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:00.259+0000 7f6dcc450640 1 -- 172.21.15.142:0/602003082 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6dc41073f0 msgr2=0x7f6dc4107830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:30:00.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:00.259+0000 7f6dcc450640 1 --2- 172.21.15.142:0/602003082 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6dc41073f0 0x7f6dc4107830 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f6db4009900 tx=0x7f6db402a410 comp rx=0 tx=0).stop 2024-09-06T14:30:00.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:00.259+0000 7f6dcc450640 1 -- 172.21.15.142:0/602003082 shutdown_connections 2024-09-06T14:30:00.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:00.259+0000 7f6dcc450640 1 --2- 172.21.15.142:0/602003082 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6dc4107d70 0x7f6dc410e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:30:00.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:00.259+0000 7f6dcc450640 1 --2- 172.21.15.142:0/602003082 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6dc41073f0 0x7f6dc4107830 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:30:00.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:00.259+0000 7f6dcc450640 1 --2- 172.21.15.142:0/602003082 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6dc41019e0 0x7f6dc4101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:30:00.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:00.259+0000 7f6dcc450640 1 -- 172.21.15.142:0/602003082 >> 172.21.15.142:0/602003082 conn(0x7f6dc40fd450 msgr2=0x7f6dc40ff870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:30:00.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:00.259+0000 7f6dcc450640 1 -- 172.21.15.142:0/602003082 shutdown_connections 2024-09-06T14:30:00.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:00.259+0000 7f6dcc450640 1 -- 172.21.15.142:0/602003082 wait complete. 2024-09-06T14:30:00.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:00.259+0000 7f6dcc450640 1 Processor -- start 2024-09-06T14:30:00.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:00.259+0000 7f6dcc450640 1 -- start start 2024-09-06T14:30:00.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:00.260+0000 7f6dcc450640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6dc41019e0 0x7f6dc41994b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:30:00.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:00.260+0000 7f6dcc450640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6dc41073f0 0x7f6dc41999f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:30:00.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:00.260+0000 7f6dcc450640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6dc4107d70 0x7f6dc41a4530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:30:00.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:00.260+0000 7f6dcc450640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f6dc4111140 con 0x7f6dc41019e0 2024-09-06T14:30:00.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:00.260+0000 7f6dcc450640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6dc4110fc0 con 0x7f6dc4107d70 2024-09-06T14:30:00.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:00.260+0000 7f6dcc450640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7f6dc41112c0 con 0x7f6dc41073f0 2024-09-06T14:30:00.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:00.260+0000 7f6dca1ec640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6dc41019e0 0x7f6dc41994b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:30:00.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:00.260+0000 7f6dc99eb640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6dc41073f0 0x7f6dc41999f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:30:00.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:00.260+0000 7f6dca1ec640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6dc41019e0 0x7f6dc41994b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:50532/0 (socket says 172.21.15.142:50532) 2024-09-06T14:30:00.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:00.260+0000 7f6dca1ec640 1 -- 172.21.15.142:0/388608186 learned_addr learned my addr 172.21.15.142:0/388608186 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:30:00.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:00.260+0000 7f6dca1ec640 1 -- 172.21.15.142:0/388608186 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6dc41073f0 msgr2=0x7f6dc41999f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:30:00.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:00.260+0000 7f6dca1ec640 1 --2- 172.21.15.142:0/388608186 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6dc41073f0 0x7f6dc41999f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:30:00.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:00.260+0000 7f6dca1ec640 1 -- 172.21.15.142:0/388608186 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6dc4107d70 msgr2=0x7f6dc41a4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:30:00.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:00.260+0000 7f6dca9ed640 1 --2- 172.21.15.142:0/388608186 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6dc4107d70 0x7f6dc41a4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:30:00.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:00.260+0000 7f6dca1ec640 1 --2- 172.21.15.142:0/388608186 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6dc4107d70 0x7f6dc41a4530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:30:00.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:00.260+0000 7f6dca1ec640 1 -- 172.21.15.142:0/388608186 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6dc41a4a70 con 0x7f6dc41019e0 2024-09-06T14:30:00.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:00.260+0000 7f6dc99eb640 1 --2- 172.21.15.142:0/388608186 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6dc41073f0 0x7f6dc41999f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-06T14:30:00.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:00.260+0000 7f6dca1ec640 1 --2- 172.21.15.142:0/388608186 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6dc41019e0 0x7f6dc41994b0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f6dc000ea60 tx=0x7f6dc000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:30:00.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:00.260+0000 7f6dbb7fe640 1 -- 172.21.15.142:0/388608186 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dc000cde0 con 0x7f6dc41019e0 2024-09-06T14:30:00.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:00.260+0000 7f6dbb7fe640 1 -- 172.21.15.142:0/388608186 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7f6dc00035f0 con 0x7f6dc41019e0 2024-09-06T14:30:00.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:00.260+0000 7f6dbb7fe640 1 -- 172.21.15.142:0/388608186 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dc00044a0 con 0x7f6dc41019e0 2024-09-06T14:30:00.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:00.261+0000 7f6dcc450640 1 -- 172.21.15.142:0/388608186 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6dc41a4c40 con 0x7f6dc41019e0 2024-09-06T14:30:00.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:00.261+0000 7f6dcc450640 1 -- 172.21.15.142:0/388608186 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6dc41a50a0 con 0x7f6dc41019e0 2024-09-06T14:30:00.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:00.261+0000 7f6dcc450640 1 -- 172.21.15.142:0/388608186 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6dc4101dc0 con 0x7f6dc41019e0 2024-09-06T14:30:00.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:00.262+0000 7f6dbb7fe640 1 -- 172.21.15.142:0/388608186 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 22) ==== 99844+0+0 (secure 0 0 0) 0x7f6dc0004000 con 0x7f6dc41019e0 2024-09-06T14:30:00.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:00.263+0000 7f6dbb7fe640 1 --2- 172.21.15.142:0/388608186 >> v2:172.21.15.142:6800/2944545612 conn(0x7f6db0077620 0x7f6db0079ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:30:00.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:00.263+0000 7f6dbb7fe640 1 -- 172.21.15.142:0/388608186 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 6130+0+0 (secure 0 0 0) 0x7f6dc0099c30 con 0x7f6dc41019e0 2024-09-06T14:30:00.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:00.263+0000 7f6dc99eb640 1 --2- 172.21.15.142:0/388608186 >> v2:172.21.15.142:6800/2944545612 conn(0x7f6db0077620 0x7f6db0079ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:30:00.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:00.263+0000 7f6dc99eb640 1 --2- 172.21.15.142:0/388608186 >> v2:172.21.15.142:6800/2944545612 conn(0x7f6db0077620 0x7f6db0079ae0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f6db4031440 tx=0x7f6db4031850 comp rx=0 tx=0).ready entity=mgr.14601 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:30:00.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:00.266+0000 7f6dbb7fe640 1 -- 172.21.15.142:0/388608186 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7f6dc0062d70 con 0x7f6dc41019e0 2024-09-06T14:30:00.453 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:00.455+0000 7f6dcc450640 1 -- 172.21.15.142:0/388608186 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f6dc402e370 con 0x7f6dc41019e0 2024-09-06T14:30:00.454 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:00.455+0000 7f6dbb7fe640 1 -- 172.21.15.142:0/388608186 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+212 (secure 0 0 0) 0x7f6dc006b220 con 0x7f6dc41019e0 2024-09-06T14:30:00.454 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:30:00.454 INFO:teuthology.orchestra.run.smithi142.stdout:{"status":"HEALTH_WARN","checks":{"MGR_MODULE_DEPENDENCY":{"severity":"HEALTH_WARN","summary":{"message":"Module 'volumes' has failed dependency: No module named 'ceph.fs'","count":1},"muted":false}},"mutes":[]} 2024-09-06T14:30:00.456 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:00.457+0000 7f6dcc450640 1 -- 172.21.15.142:0/388608186 >> v2:172.21.15.142:6800/2944545612 conn(0x7f6db0077620 msgr2=0x7f6db0079ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:30:00.456 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:00.457+0000 7f6dcc450640 1 --2- 172.21.15.142:0/388608186 >> v2:172.21.15.142:6800/2944545612 conn(0x7f6db0077620 0x7f6db0079ae0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f6db4031440 tx=0x7f6db4031850 comp rx=0 tx=0).stop 2024-09-06T14:30:00.456 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:00.457+0000 7f6dcc450640 1 -- 172.21.15.142:0/388608186 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6dc41019e0 msgr2=0x7f6dc41994b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:30:00.456 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:00.457+0000 7f6dcc450640 1 --2- 172.21.15.142:0/388608186 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6dc41019e0 0x7f6dc41994b0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f6dc000ea60 tx=0x7f6dc000ef20 comp rx=0 tx=0).stop 2024-09-06T14:30:00.456 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:00.457+0000 7f6dcc450640 1 -- 172.21.15.142:0/388608186 shutdown_connections 2024-09-06T14:30:00.456 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:00.457+0000 7f6dcc450640 1 --2- 172.21.15.142:0/388608186 >> v2:172.21.15.142:6800/2944545612 conn(0x7f6db0077620 0x7f6db0079ae0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:30:00.456 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:00.457+0000 7f6dcc450640 1 --2- 172.21.15.142:0/388608186 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6dc4107d70 0x7f6dc41a4530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:30:00.456 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:00.457+0000 7f6dcc450640 1 --2- 172.21.15.142:0/388608186 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6dc41073f0 0x7f6dc41999f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:30:00.456 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:00.457+0000 7f6dcc450640 1 --2- 172.21.15.142:0/388608186 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6dc41019e0 0x7f6dc41994b0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:30:00.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:00.457+0000 7f6dcc450640 1 -- 172.21.15.142:0/388608186 >> 172.21.15.142:0/388608186 conn(0x7f6dc40fd450 msgr2=0x7f6dc4104a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:30:00.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:00.457+0000 7f6dcc450640 1 -- 172.21.15.142:0/388608186 shutdown_connections 2024-09-06T14:30:00.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:00.458+0000 7f6dcc450640 1 -- 172.21.15.142:0/388608186 wait complete. 2024-09-06T14:30:01.048 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:30:00 smithi162 ceph-mon[33787]: pgmap v157: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:30:01.048 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:30:00 smithi162 ceph-mon[33787]: overall HEALTH_WARN Module 'volumes' has failed dependency: No module named 'ceph.fs' 2024-09-06T14:30:01.048 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:30:00 smithi162 ceph-mon[33787]: from='client.? 172.21.15.142:0/388608186' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:30:01.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:30:00 smithi142 ceph-mon[37372]: pgmap v157: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:30:01.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:30:00 smithi142 ceph-mon[37372]: overall HEALTH_WARN Module 'volumes' has failed dependency: No module named 'ceph.fs' 2024-09-06T14:30:01.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:30:00 smithi142 ceph-mon[37372]: from='client.? 172.21.15.142:0/388608186' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:30:01.210 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:30:00 smithi142 ceph-mon[29958]: pgmap v157: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:30:01.210 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:30:00 smithi142 ceph-mon[29958]: overall HEALTH_WARN Module 'volumes' has failed dependency: No module named 'ceph.fs' 2024-09-06T14:30:01.210 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:30:00 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/388608186' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:30:03.048 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:30:02 smithi162 ceph-mon[33787]: pgmap v158: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:30:03.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:30:02 smithi142 ceph-mon[29958]: pgmap v158: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:30:03.210 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:30:02 smithi142 ceph-mon[37372]: pgmap v158: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:30:04.244 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph health --format=json 2024-09-06T14:30:04.506 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:30:04.627 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:30:04 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: ::ffff:172.21.15.162 - - [06/Sep/2024:14:30:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-06T14:30:04.888 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:30:04 smithi142 ceph-mon[29958]: pgmap v159: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:30:04.888 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:30:04 smithi142 ceph-mon[37372]: pgmap v159: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:30:05.012 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:04.970+0000 7f7ac5d03640 1 -- 172.21.15.142:0/3889120065 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7ac010f110 msgr2=0x7f7ac0111500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:30:05.012 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:04.970+0000 7f7ac5d03640 1 --2- 172.21.15.142:0/3889120065 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7ac010f110 0x7f7ac0111500 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f7ab8009ca0 tx=0x7f7ab802a8c0 comp rx=0 tx=0).stop 2024-09-06T14:30:05.013 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:04.970+0000 7f7ac5d03640 1 -- 172.21.15.142:0/3889120065 shutdown_connections 2024-09-06T14:30:05.013 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:04.970+0000 7f7ac5d03640 1 --2- 172.21.15.142:0/3889120065 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7ac010f110 0x7f7ac0111500 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:30:05.013 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:04.970+0000 7f7ac5d03640 1 --2- 172.21.15.142:0/3889120065 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7ac0075b40 0x7f7ac010ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:30:05.013 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:04.970+0000 7f7ac5d03640 1 --2- 172.21.15.142:0/3889120065 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7ac0075220 0x7f7ac0075600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:30:05.013 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:04.970+0000 7f7ac5d03640 1 -- 172.21.15.142:0/3889120065 >> 172.21.15.142:0/3889120065 conn(0x7f7ac00fe570 msgr2=0x7f7ac0100990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:30:05.013 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:04.970+0000 7f7ac5d03640 1 -- 172.21.15.142:0/3889120065 shutdown_connections 2024-09-06T14:30:05.013 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:04.970+0000 7f7ac5d03640 1 -- 172.21.15.142:0/3889120065 wait complete. 2024-09-06T14:30:05.013 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:04.971+0000 7f7ac5d03640 1 Processor -- start 2024-09-06T14:30:05.013 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:04.971+0000 7f7ac5d03640 1 -- start start 2024-09-06T14:30:05.013 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:04.971+0000 7f7ac5d03640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7ac0075220 0x7f7ac019aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:30:05.013 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:04.971+0000 7f7ac5d03640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7ac0075b40 0x7f7ac019af70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:30:05.013 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:04.971+0000 7f7ac5d03640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7ac010f110 0x7f7ac01a0310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:30:05.013 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:04.971+0000 7f7ac5d03640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f7ac0114400 con 0x7f7ac0075220 2024-09-06T14:30:05.014 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:04.971+0000 7f7ac5d03640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7ac0114280 con 0x7f7ac010f110 2024-09-06T14:30:05.014 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:04.971+0000 7f7ac5d03640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7f7ac0114580 con 0x7f7ac0075b40 2024-09-06T14:30:05.014 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:04.971+0000 7f7abeffd640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7ac0075b40 0x7f7ac019af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:30:05.014 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:04.971+0000 7f7abf7fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7ac0075220 0x7f7ac019aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:30:05.014 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:04.971+0000 7f7abeffd640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7ac0075b40 0x7f7ac019af70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:50670/0 (socket says 172.21.15.142:50670) 2024-09-06T14:30:05.014 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:04.971+0000 7f7abeffd640 1 -- 172.21.15.142:0/3178826238 learned_addr learned my addr 172.21.15.142:0/3178826238 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:30:05.014 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:04.971+0000 7f7abffff640 1 --2- 172.21.15.142:0/3178826238 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7ac010f110 0x7f7ac01a0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:30:05.014 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:04.971+0000 7f7abeffd640 1 -- 172.21.15.142:0/3178826238 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7ac010f110 msgr2=0x7f7ac01a0310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:30:05.014 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:04.971+0000 7f7abeffd640 1 --2- 172.21.15.142:0/3178826238 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7ac010f110 0x7f7ac01a0310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:30:05.014 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:04.971+0000 7f7abeffd640 1 -- 172.21.15.142:0/3178826238 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7ac0075220 msgr2=0x7f7ac019aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:30:05.014 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:04.971+0000 7f7abeffd640 1 --2- 172.21.15.142:0/3178826238 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7ac0075220 0x7f7ac019aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:30:05.014 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:04.971+0000 7f7abeffd640 1 -- 172.21.15.142:0/3178826238 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ac01a0ae0 con 0x7f7ac0075b40 2024-09-06T14:30:05.014 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:04.971+0000 7f7abf7fe640 1 --2- 172.21.15.142:0/3178826238 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7ac0075220 0x7f7ac019aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-06T14:30:05.014 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:04.972+0000 7f7abeffd640 1 --2- 172.21.15.142:0/3178826238 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7ac0075b40 0x7f7ac019af70 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f7ab400ea60 tx=0x7f7ab400ee60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:30:05.015 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:04.972+0000 7f7abcff9640 1 -- 172.21.15.142:0/3178826238 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ab400cd20 con 0x7f7ac0075b40 2024-09-06T14:30:05.015 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:04.972+0000 7f7abcff9640 1 -- 172.21.15.142:0/3178826238 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7f7ab40035f0 con 0x7f7ac0075b40 2024-09-06T14:30:05.015 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:04.972+0000 7f7ac5d03640 1 -- 172.21.15.142:0/3178826238 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ac01a0dd0 con 0x7f7ac0075b40 2024-09-06T14:30:05.015 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:04.972+0000 7f7abcff9640 1 -- 172.21.15.142:0/3178826238 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ab40044a0 con 0x7f7ac0075b40 2024-09-06T14:30:05.015 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:04.972+0000 7f7ac5d03640 1 -- 172.21.15.142:0/3178826238 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ac01a7490 con 0x7f7ac0075b40 2024-09-06T14:30:05.015 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:04.973+0000 7f7ac5d03640 1 -- 172.21.15.142:0/3178826238 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ac0075600 con 0x7f7ac0075b40 2024-09-06T14:30:05.015 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:04.973+0000 7f7abcff9640 1 -- 172.21.15.142:0/3178826238 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 22) ==== 99844+0+0 (secure 0 0 0) 0x7f7ab4004000 con 0x7f7ac0075b40 2024-09-06T14:30:05.015 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:04.974+0000 7f7abcff9640 1 --2- 172.21.15.142:0/3178826238 >> v2:172.21.15.142:6800/2944545612 conn(0x7f7aa8077620 0x7f7aa8079ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:30:05.015 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:04.974+0000 7f7abcff9640 1 -- 172.21.15.142:0/3178826238 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(64..64 src has 1..64) ==== 6130+0+0 (secure 0 0 0) 0x7f7ab4099d90 con 0x7f7ac0075b40 2024-09-06T14:30:05.015 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:04.974+0000 7f7abf7fe640 1 --2- 172.21.15.142:0/3178826238 >> v2:172.21.15.142:6800/2944545612 conn(0x7f7aa8077620 0x7f7aa8079ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:30:05.015 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:04.974+0000 7f7abf7fe640 1 --2- 172.21.15.142:0/3178826238 >> v2:172.21.15.142:6800/2944545612 conn(0x7f7aa8077620 0x7f7aa8079ae0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f7ab0009d60 tx=0x7f7ab0009110 comp rx=0 tx=0).ready entity=mgr.14601 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:30:05.015 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:04.978+0000 7f7abcff9640 1 -- 172.21.15.142:0/3178826238 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7f7ab409d030 con 0x7f7ac0075b40 2024-09-06T14:30:05.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:30:04 smithi162 ceph-mon[33787]: pgmap v159: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:30:05.160 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:05.163+0000 7f7ac5d03640 1 -- 172.21.15.142:0/3178826238 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f7ac01099a0 con 0x7f7ac0075b40 2024-09-06T14:30:05.161 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:05.163+0000 7f7abcff9640 1 -- 172.21.15.142:0/3178826238 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+212 (secure 0 0 0) 0x7f7ab4062e50 con 0x7f7ac0075b40 2024-09-06T14:30:05.161 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:30:05.161 INFO:teuthology.orchestra.run.smithi142.stdout:{"status":"HEALTH_WARN","checks":{"MGR_MODULE_DEPENDENCY":{"severity":"HEALTH_WARN","summary":{"message":"Module 'volumes' has failed dependency: No module named 'ceph.fs'","count":1},"muted":false}},"mutes":[]} 2024-09-06T14:30:05.163 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:05.165+0000 7f7ac5d03640 1 -- 172.21.15.142:0/3178826238 >> v2:172.21.15.142:6800/2944545612 conn(0x7f7aa8077620 msgr2=0x7f7aa8079ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:30:05.163 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:05.165+0000 7f7ac5d03640 1 --2- 172.21.15.142:0/3178826238 >> v2:172.21.15.142:6800/2944545612 conn(0x7f7aa8077620 0x7f7aa8079ae0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f7ab0009d60 tx=0x7f7ab0009110 comp rx=0 tx=0).stop 2024-09-06T14:30:05.164 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:05.165+0000 7f7ac5d03640 1 -- 172.21.15.142:0/3178826238 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7ac0075b40 msgr2=0x7f7ac019af70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:30:05.164 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:05.165+0000 7f7ac5d03640 1 --2- 172.21.15.142:0/3178826238 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7ac0075b40 0x7f7ac019af70 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f7ab400ea60 tx=0x7f7ab400ee60 comp rx=0 tx=0).stop 2024-09-06T14:30:05.164 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:05.166+0000 7f7ac5d03640 1 -- 172.21.15.142:0/3178826238 shutdown_connections 2024-09-06T14:30:05.164 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:05.166+0000 7f7ac5d03640 1 --2- 172.21.15.142:0/3178826238 >> v2:172.21.15.142:6800/2944545612 conn(0x7f7aa8077620 0x7f7aa8079ae0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:30:05.164 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:05.166+0000 7f7ac5d03640 1 --2- 172.21.15.142:0/3178826238 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7ac010f110 0x7f7ac01a0310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:30:05.164 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:05.166+0000 7f7ac5d03640 1 --2- 172.21.15.142:0/3178826238 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7ac0075b40 0x7f7ac019af70 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:30:05.164 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:05.166+0000 7f7ac5d03640 1 --2- 172.21.15.142:0/3178826238 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7ac0075220 0x7f7ac019aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:30:05.164 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:05.166+0000 7f7ac5d03640 1 -- 172.21.15.142:0/3178826238 >> 172.21.15.142:0/3178826238 conn(0x7f7ac00fe570 msgr2=0x7f7ac0106b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:30:05.164 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:05.166+0000 7f7ac5d03640 1 -- 172.21.15.142:0/3178826238 shutdown_connections 2024-09-06T14:30:05.164 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:05.166+0000 7f7ac5d03640 1 -- 172.21.15.142:0/3178826238 wait complete. 2024-09-06T14:30:05.959 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:30:05 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/3178826238' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:30:05.959 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:30:05 smithi142 ceph-mon[37372]: from='client.? 172.21.15.142:0/3178826238' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:30:06.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:30:05 smithi162 ceph-mon[33787]: from='client.? 172.21.15.142:0/3178826238' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:30:07.048 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:30:06 smithi162 ceph-mon[33787]: pgmap v160: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:30:07.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:30:06 smithi142 ceph-mon[29958]: pgmap v160: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:30:07.210 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:30:06 smithi142 ceph-mon[37372]: pgmap v160: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-06T14:30:07.797 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 06 14:30:07 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-iscsi-iscsi-a[59853]: debug there is no tcmu-runner data available 2024-09-06T14:30:08.993 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph health --format=json 2024-09-06T14:30:09.021 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:30:08 smithi142 ceph-mon[29958]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-06T14:30:09.021 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:30:08 smithi142 ceph-mon[29958]: pgmap v161: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:30:09.021 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:30:08 smithi142 ceph-mon[37372]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-06T14:30:09.022 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:30:08 smithi142 ceph-mon[37372]: pgmap v161: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:30:09.048 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:30:08 smithi162 ceph-mon[33787]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-06T14:30:09.048 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:30:08 smithi162 ceph-mon[33787]: pgmap v161: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:30:09.256 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:30:09.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:09.801+0000 7f26e5a63640 1 -- 172.21.15.142:0/679662522 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f26e01019c0 msgr2=0x7f26e0101da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:30:09.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:09.801+0000 7f26e5a63640 1 --2- 172.21.15.142:0/679662522 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f26e01019c0 0x7f26e0101da0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f26d0009ca0 tx=0x7f26d002a8c0 comp rx=0 tx=0).stop 2024-09-06T14:30:09.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:09.802+0000 7f26e5a63640 1 -- 172.21.15.142:0/679662522 shutdown_connections 2024-09-06T14:30:09.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:09.802+0000 7f26e5a63640 1 --2- 172.21.15.142:0/679662522 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f26e0107d50 0x7f26e010e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:30:09.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:09.802+0000 7f26e5a63640 1 --2- 172.21.15.142:0/679662522 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f26e01073d0 0x7f26e0107810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:30:09.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:09.802+0000 7f26e5a63640 1 --2- 172.21.15.142:0/679662522 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f26e01019c0 0x7f26e0101da0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:30:09.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:09.802+0000 7f26e5a63640 1 -- 172.21.15.142:0/679662522 >> 172.21.15.142:0/679662522 conn(0x7f26e00fd430 msgr2=0x7f26e00ff850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:30:09.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:09.802+0000 7f26e5a63640 1 -- 172.21.15.142:0/679662522 shutdown_connections 2024-09-06T14:30:09.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:09.802+0000 7f26e5a63640 1 -- 172.21.15.142:0/679662522 wait complete. 2024-09-06T14:30:09.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:09.802+0000 7f26e5a63640 1 Processor -- start 2024-09-06T14:30:09.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:09.802+0000 7f26e5a63640 1 -- start start 2024-09-06T14:30:09.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:09.803+0000 7f26e5a63640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f26e01019c0 0x7f26e0197610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:30:09.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:09.803+0000 7f26e5a63640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f26e01073d0 0x7f26e0197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:30:09.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:09.803+0000 7f26e5a63640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f26e0107d50 0x7f26e019cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:30:09.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:09.803+0000 7f26e5a63640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f26e0111120 con 0x7f26e01073d0 2024-09-06T14:30:09.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:09.803+0000 7f26e5a63640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f26e0110fa0 con 0x7f26e0107d50 2024-09-06T14:30:09.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:09.803+0000 7f26e5a63640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7f26e01112a0 con 0x7f26e01019c0 2024-09-06T14:30:09.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:09.803+0000 7f26df7fe640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f26e01019c0 0x7f26e0197610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:30:09.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:09.803+0000 7f26deffd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f26e01073d0 0x7f26e0197b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:30:09.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:09.803+0000 7f26df7fe640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f26e01019c0 0x7f26e0197610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:38024/0 (socket says 172.21.15.142:38024) 2024-09-06T14:30:09.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:09.803+0000 7f26df7fe640 1 -- 172.21.15.142:0/2403573143 learned_addr learned my addr 172.21.15.142:0/2403573143 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:30:09.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:09.803+0000 7f26dffff640 1 --2- 172.21.15.142:0/2403573143 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f26e0107d50 0x7f26e019cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:30:09.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:09.803+0000 7f26deffd640 1 -- 172.21.15.142:0/2403573143 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f26e01019c0 msgr2=0x7f26e0197610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:30:09.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:09.803+0000 7f26deffd640 1 --2- 172.21.15.142:0/2403573143 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f26e01019c0 0x7f26e0197610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:30:09.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:09.803+0000 7f26deffd640 1 -- 172.21.15.142:0/2403573143 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f26e0107d50 msgr2=0x7f26e019cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:30:09.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:09.803+0000 7f26deffd640 1 --2- 172.21.15.142:0/2403573143 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f26e0107d50 0x7f26e019cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:30:09.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:09.803+0000 7f26deffd640 1 -- 172.21.15.142:0/2403573143 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26e019d6c0 con 0x7f26e01073d0 2024-09-06T14:30:09.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:09.803+0000 7f26dffff640 1 --2- 172.21.15.142:0/2403573143 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f26e0107d50 0x7f26e019cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-06T14:30:09.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:09.803+0000 7f26df7fe640 1 --2- 172.21.15.142:0/2403573143 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f26e01019c0 0x7f26e0197610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-06T14:30:09.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:09.803+0000 7f26deffd640 1 --2- 172.21.15.142:0/2403573143 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f26e01073d0 0x7f26e0197b50 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f26c800e9f0 tx=0x7f26c800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:30:09.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:09.803+0000 7f26dcff9640 1 -- 172.21.15.142:0/2403573143 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26c800cd80 con 0x7f26e01073d0 2024-09-06T14:30:09.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:09.804+0000 7f26e5a63640 1 -- 172.21.15.142:0/2403573143 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26e019d9b0 con 0x7f26e01073d0 2024-09-06T14:30:09.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:09.804+0000 7f26dcff9640 1 -- 172.21.15.142:0/2403573143 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7f26c80035f0 con 0x7f26e01073d0 2024-09-06T14:30:09.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:09.804+0000 7f26dcff9640 1 -- 172.21.15.142:0/2403573143 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26c80044a0 con 0x7f26e01073d0 2024-09-06T14:30:09.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:09.804+0000 7f26e5a63640 1 -- 172.21.15.142:0/2403573143 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26e01a4240 con 0x7f26e01073d0 2024-09-06T14:30:09.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:09.804+0000 7f26e5a63640 1 -- 172.21.15.142:0/2403573143 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26e0101da0 con 0x7f26e01073d0 2024-09-06T14:30:09.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:09.805+0000 7f26dcff9640 1 -- 172.21.15.142:0/2403573143 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 22) ==== 99844+0+0 (secure 0 0 0) 0x7f26c8004000 con 0x7f26e01073d0 2024-09-06T14:30:09.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:09.805+0000 7f26dcff9640 1 --2- 172.21.15.142:0/2403573143 >> v2:172.21.15.142:6800/2944545612 conn(0x7f26cc077620 0x7f26cc079ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:30:09.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:09.806+0000 7f26df7fe640 1 --2- 172.21.15.142:0/2403573143 >> v2:172.21.15.142:6800/2944545612 conn(0x7f26cc077620 0x7f26cc079ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:30:09.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:09.806+0000 7f26dcff9640 1 -- 172.21.15.142:0/2403573143 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 6130+0+0 (secure 0 0 0) 0x7f26c8099ce0 con 0x7f26e01073d0 2024-09-06T14:30:09.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:09.806+0000 7f26df7fe640 1 --2- 172.21.15.142:0/2403573143 >> v2:172.21.15.142:6800/2944545612 conn(0x7f26cc077620 0x7f26cc079ae0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f26d0005fe0 tx=0x7f26d0005f50 comp rx=0 tx=0).ready entity=mgr.14601 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:30:09.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:09.809+0000 7f26dcff9640 1 -- 172.21.15.142:0/2403573143 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7f26c8062da0 con 0x7f26e01073d0 2024-09-06T14:30:09.995 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:09.997+0000 7f26e5a63640 1 -- 172.21.15.142:0/2403573143 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f26e0109b30 con 0x7f26e01073d0 2024-09-06T14:30:09.995 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:09.998+0000 7f26dcff9640 1 -- 172.21.15.142:0/2403573143 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+212 (secure 0 0 0) 0x7f26c806b250 con 0x7f26e01073d0 2024-09-06T14:30:09.996 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:30:09.996 INFO:teuthology.orchestra.run.smithi142.stdout:{"status":"HEALTH_WARN","checks":{"MGR_MODULE_DEPENDENCY":{"severity":"HEALTH_WARN","summary":{"message":"Module 'volumes' has failed dependency: No module named 'ceph.fs'","count":1},"muted":false}},"mutes":[]} 2024-09-06T14:30:09.998 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:10.000+0000 7f26e5a63640 1 -- 172.21.15.142:0/2403573143 >> v2:172.21.15.142:6800/2944545612 conn(0x7f26cc077620 msgr2=0x7f26cc079ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:30:09.998 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:10.000+0000 7f26e5a63640 1 --2- 172.21.15.142:0/2403573143 >> v2:172.21.15.142:6800/2944545612 conn(0x7f26cc077620 0x7f26cc079ae0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f26d0005fe0 tx=0x7f26d0005f50 comp rx=0 tx=0).stop 2024-09-06T14:30:09.998 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:10.000+0000 7f26e5a63640 1 -- 172.21.15.142:0/2403573143 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f26e01073d0 msgr2=0x7f26e0197b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:30:09.998 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:10.000+0000 7f26e5a63640 1 --2- 172.21.15.142:0/2403573143 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f26e01073d0 0x7f26e0197b50 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f26c800e9f0 tx=0x7f26c800eeb0 comp rx=0 tx=0).stop 2024-09-06T14:30:09.998 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:10.000+0000 7f26e5a63640 1 -- 172.21.15.142:0/2403573143 shutdown_connections 2024-09-06T14:30:09.998 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:10.000+0000 7f26e5a63640 1 --2- 172.21.15.142:0/2403573143 >> v2:172.21.15.142:6800/2944545612 conn(0x7f26cc077620 0x7f26cc079ae0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:30:09.998 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:10.000+0000 7f26e5a63640 1 --2- 172.21.15.142:0/2403573143 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f26e0107d50 0x7f26e019cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:30:09.998 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:10.000+0000 7f26e5a63640 1 --2- 172.21.15.142:0/2403573143 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f26e01073d0 0x7f26e0197b50 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:30:09.998 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:10.000+0000 7f26e5a63640 1 --2- 172.21.15.142:0/2403573143 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f26e01019c0 0x7f26e0197610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:30:09.999 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:10.000+0000 7f26e5a63640 1 -- 172.21.15.142:0/2403573143 >> 172.21.15.142:0/2403573143 conn(0x7f26e00fd430 msgr2=0x7f26e00ff820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:30:09.999 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:10.000+0000 7f26e5a63640 1 -- 172.21.15.142:0/2403573143 shutdown_connections 2024-09-06T14:30:09.999 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:10.000+0000 7f26e5a63640 1 -- 172.21.15.142:0/2403573143 wait complete. 2024-09-06T14:30:11.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:30:10 smithi162 ceph-mon[33787]: pgmap v162: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-06T14:30:11.048 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:30:10 smithi162 ceph-mon[33787]: from='client.? 172.21.15.142:0/2403573143' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:30:11.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:30:10 smithi142 ceph-mon[37372]: pgmap v162: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-06T14:30:11.210 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:30:10 smithi142 ceph-mon[37372]: from='client.? 172.21.15.142:0/2403573143' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:30:11.210 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:30:10 smithi142 ceph-mon[29958]: pgmap v162: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-06T14:30:11.210 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:30:10 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/2403573143' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:30:12.048 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:30:11 smithi162 ceph-mon[33787]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-06T14:30:12.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:30:11 smithi142 ceph-mon[29958]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-06T14:30:12.210 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:30:11 smithi142 ceph-mon[37372]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-06T14:30:13.048 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:30:12 smithi162 ceph-mon[33787]: pgmap v163: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:30:13.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:30:12 smithi142 ceph-mon[29958]: pgmap v163: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:30:13.210 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:30:12 smithi142 ceph-mon[37372]: pgmap v163: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-06T14:30:13.761 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73 shell --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 -- ceph health --format=json 2024-09-06T14:30:14.019 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/mon.a/config 2024-09-06T14:30:14.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:14.511+0000 7f6295698640 1 -- 172.21.15.142:0/3822686761 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6290107d50 msgr2=0x7f629010e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:30:14.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:14.511+0000 7f6295698640 1 --2- 172.21.15.142:0/3822686761 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6290107d50 0x7f629010e550 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f6288009be0 tx=0x7f628802a6c0 comp rx=0 tx=0).stop 2024-09-06T14:30:14.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:14.511+0000 7f6295698640 1 -- 172.21.15.142:0/3822686761 shutdown_connections 2024-09-06T14:30:14.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:14.511+0000 7f6295698640 1 --2- 172.21.15.142:0/3822686761 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6290107d50 0x7f629010e550 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:30:14.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:14.511+0000 7f6295698640 1 --2- 172.21.15.142:0/3822686761 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f62901073d0 0x7f6290107810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:30:14.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:14.511+0000 7f6295698640 1 --2- 172.21.15.142:0/3822686761 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f62901019c0 0x7f6290101da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:30:14.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:14.511+0000 7f6295698640 1 -- 172.21.15.142:0/3822686761 >> 172.21.15.142:0/3822686761 conn(0x7f62900fd430 msgr2=0x7f62900ff850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:30:14.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:14.511+0000 7f6295698640 1 -- 172.21.15.142:0/3822686761 shutdown_connections 2024-09-06T14:30:14.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:14.511+0000 7f6295698640 1 -- 172.21.15.142:0/3822686761 wait complete. 2024-09-06T14:30:14.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:14.512+0000 7f6295698640 1 Processor -- start 2024-09-06T14:30:14.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:14.512+0000 7f6295698640 1 -- start start 2024-09-06T14:30:14.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:14.512+0000 7f6295698640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f62901019c0 0x7f62900fd930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:30:14.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:14.512+0000 7f6295698640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f62901073d0 0x7f62900fde70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:30:14.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:14.512+0000 7f6295698640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6290107d50 0x7f62900fe3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:30:14.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:14.512+0000 7f6295698640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f62901110e0 con 0x7f6290107d50 2024-09-06T14:30:14.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:14.512+0000 7f6295698640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6290110f60 con 0x7f62901073d0 2024-09-06T14:30:14.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:14.512+0000 7f6295698640 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 -- 0x7f6290111260 con 0x7f62901019c0 2024-09-06T14:30:14.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:14.512+0000 7f628effd640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f62901019c0 0x7f62900fd930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:30:14.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:14.512+0000 7f628effd640 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f62901019c0 0x7f62900fd930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:38036/0 (socket says 172.21.15.142:38036) 2024-09-06T14:30:14.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:14.512+0000 7f628effd640 1 -- 172.21.15.142:0/300733096 learned_addr learned my addr 172.21.15.142:0/300733096 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-06T14:30:14.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:14.512+0000 7f628f7fe640 1 --2- 172.21.15.142:0/300733096 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6290107d50 0x7f62900fe3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:30:14.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:14.512+0000 7f628e7fc640 1 --2- 172.21.15.142:0/300733096 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f62901073d0 0x7f62900fde70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:30:14.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:14.513+0000 7f628effd640 1 -- 172.21.15.142:0/300733096 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f62901073d0 msgr2=0x7f62900fde70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:30:14.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:14.513+0000 7f628effd640 1 --2- 172.21.15.142:0/300733096 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f62901073d0 0x7f62900fde70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:30:14.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:14.513+0000 7f628effd640 1 -- 172.21.15.142:0/300733096 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6290107d50 msgr2=0x7f62900fe3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:30:14.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:14.513+0000 7f628effd640 1 --2- 172.21.15.142:0/300733096 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6290107d50 0x7f62900fe3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:30:14.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:14.513+0000 7f628effd640 1 -- 172.21.15.142:0/300733096 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62901a6a80 con 0x7f62901019c0 2024-09-06T14:30:14.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:14.513+0000 7f628effd640 1 --2- 172.21.15.142:0/300733096 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f62901019c0 0x7f62900fd930 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f628000e9c0 tx=0x7f628000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:30:14.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:14.513+0000 7f6277fff640 1 -- 172.21.15.142:0/300733096 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f628000cde0 con 0x7f62901019c0 2024-09-06T14:30:14.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:14.513+0000 7f6277fff640 1 -- 172.21.15.142:0/300733096 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(25 keys) ==== 1054+0+0 (secure 0 0 0) 0x7f62800035f0 con 0x7f62901019c0 2024-09-06T14:30:14.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:14.513+0000 7f6295698640 1 -- 172.21.15.142:0/300733096 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62901a6d70 con 0x7f62901019c0 2024-09-06T14:30:14.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:14.513+0000 7f6277fff640 1 -- 172.21.15.142:0/300733096 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62800044a0 con 0x7f62901019c0 2024-09-06T14:30:14.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:14.513+0000 7f6295698640 1 -- 172.21.15.142:0/300733096 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f62901a7270 con 0x7f62901019c0 2024-09-06T14:30:14.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:14.514+0000 7f6295698640 1 -- 172.21.15.142:0/300733096 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6290101da0 con 0x7f62901019c0 2024-09-06T14:30:14.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:14.515+0000 7f6277fff640 1 -- 172.21.15.142:0/300733096 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 22) ==== 99844+0+0 (secure 0 0 0) 0x7f6280004000 con 0x7f62901019c0 2024-09-06T14:30:14.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:14.515+0000 7f6277fff640 1 --2- 172.21.15.142:0/300733096 >> v2:172.21.15.142:6800/2944545612 conn(0x7f6278077620 0x7f6278079ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-06T14:30:14.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:14.515+0000 7f6277fff640 1 -- 172.21.15.142:0/300733096 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(64..64 src has 1..64) ==== 6130+0+0 (secure 0 0 0) 0x7f6280099c80 con 0x7f62901019c0 2024-09-06T14:30:14.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:14.515+0000 7f628e7fc640 1 --2- 172.21.15.142:0/300733096 >> v2:172.21.15.142:6800/2944545612 conn(0x7f6278077620 0x7f6278079ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-06T14:30:14.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:14.515+0000 7f628e7fc640 1 --2- 172.21.15.142:0/300733096 >> v2:172.21.15.142:6800/2944545612 conn(0x7f6278077620 0x7f6278079ae0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f62900ff420 tx=0x7f628400a250 comp rx=0 tx=0).ready entity=mgr.14601 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-06T14:30:14.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:14.519+0000 7f6277fff640 1 -- 172.21.15.142:0/300733096 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180031 (secure 0 0 0) 0x7f6280066e90 con 0x7f62901019c0 2024-09-06T14:30:14.709 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:30:14 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y[30184]: ::ffff:172.21.15.162 - - [06/Sep/2024:14:30:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-06T14:30:14.714 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:14.717+0000 7f6295698640 1 -- 172.21.15.142:0/300733096 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f629010aba0 con 0x7f62901019c0 2024-09-06T14:30:14.715 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:14.717+0000 7f6277fff640 1 -- 172.21.15.142:0/300733096 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+212 (secure 0 0 0) 0x7f628006b1d0 con 0x7f62901019c0 2024-09-06T14:30:14.715 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-06T14:30:14.715 INFO:teuthology.orchestra.run.smithi142.stdout:{"status":"HEALTH_WARN","checks":{"MGR_MODULE_DEPENDENCY":{"severity":"HEALTH_WARN","summary":{"message":"Module 'volumes' has failed dependency: No module named 'ceph.fs'","count":1},"muted":false}},"mutes":[]} 2024-09-06T14:30:14.717 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:14.719+0000 7f6295698640 1 -- 172.21.15.142:0/300733096 >> v2:172.21.15.142:6800/2944545612 conn(0x7f6278077620 msgr2=0x7f6278079ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:30:14.717 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:14.719+0000 7f6295698640 1 --2- 172.21.15.142:0/300733096 >> v2:172.21.15.142:6800/2944545612 conn(0x7f6278077620 0x7f6278079ae0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f62900ff420 tx=0x7f628400a250 comp rx=0 tx=0).stop 2024-09-06T14:30:14.717 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:14.719+0000 7f6295698640 1 -- 172.21.15.142:0/300733096 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f62901019c0 msgr2=0x7f62900fd930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-06T14:30:14.718 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:14.719+0000 7f6295698640 1 --2- 172.21.15.142:0/300733096 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f62901019c0 0x7f62900fd930 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f628000e9c0 tx=0x7f628000ee80 comp rx=0 tx=0).stop 2024-09-06T14:30:14.718 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:14.720+0000 7f6295698640 1 -- 172.21.15.142:0/300733096 shutdown_connections 2024-09-06T14:30:14.718 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:14.720+0000 7f6295698640 1 --2- 172.21.15.142:0/300733096 >> v2:172.21.15.142:6800/2944545612 conn(0x7f6278077620 0x7f6278079ae0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:30:14.718 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:14.720+0000 7f6295698640 1 --2- 172.21.15.142:0/300733096 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6290107d50 0x7f62900fe3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:30:14.718 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:14.720+0000 7f6295698640 1 --2- 172.21.15.142:0/300733096 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f62901073d0 0x7f62900fde70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:30:14.718 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:14.720+0000 7f6295698640 1 --2- 172.21.15.142:0/300733096 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f62901019c0 0x7f62900fd930 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-06T14:30:14.718 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:14.720+0000 7f6295698640 1 -- 172.21.15.142:0/300733096 >> 172.21.15.142:0/300733096 conn(0x7f62900fd430 msgr2=0x7f629006aa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-06T14:30:14.718 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:14.720+0000 7f6295698640 1 -- 172.21.15.142:0/300733096 shutdown_connections 2024-09-06T14:30:14.718 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-06T14:30:14.720+0000 7f6295698640 1 -- 172.21.15.142:0/300733096 wait complete. 2024-09-06T14:30:14.965 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:30:14 smithi142 ceph-mon[29958]: pgmap v164: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-06T14:30:14.965 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:30:14 smithi142 ceph-mon[29958]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-06T14:30:14.965 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:30:14 smithi142 ceph-mon[29958]: from='client.? 172.21.15.142:0/300733096' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:30:14.965 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:30:14 smithi142 ceph-mon[37372]: pgmap v164: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-06T14:30:14.966 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:30:14 smithi142 ceph-mon[37372]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-06T14:30:14.966 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:30:14 smithi142 ceph-mon[37372]: from='client.? 172.21.15.142:0/300733096' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:30:15.048 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:30:14 smithi162 ceph-mon[33787]: pgmap v164: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-06T14:30:15.048 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:30:14 smithi162 ceph-mon[33787]: from='mgr.14601 172.21.15.142:0/4157300420' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-06T14:30:15.048 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:30:14 smithi162 ceph-mon[33787]: from='client.? 172.21.15.142:0/300733096' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-06T14:30:15.491 INFO:tasks.cephadm:Teardown begin 2024-09-06T14:30:15.491 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_8b38c33cecf4b0b7a9b3116eb9089e21e444ac73/qa/tasks/cephadm.py", line 2371, in task healthy(ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_ceph-c_8b38c33cecf4b0b7a9b3116eb9089e21e444ac73/qa/tasks/ceph.py", line 1537, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/git.ceph.com_ceph-c_8b38c33cecf4b0b7a9b3116eb9089e21e444ac73/qa/tasks/ceph_manager.py", line 3299, in wait_until_healthy assert time.time() - start < timeout, \ AssertionError: timeout expired in wait_until_healthy 2024-09-06T14:30:15.494 DEBUG:teuthology.orchestra.run.smithi142:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-06T14:30:15.523 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-06T14:30:15.554 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-09-06T14:30:15.554 DEBUG:teuthology.orchestra.run.smithi142:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-06T14:30:15.580 DEBUG:teuthology.orchestra.run.smithi162:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-06T14:30:15.610 INFO:tasks.cephadm:Stopping all daemons... 2024-09-06T14:30:15.610 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-09-06T14:30:15.610 DEBUG:teuthology.orchestra.run.smithi142:> sudo systemctl stop ceph-ce32462e-6c5a-11ef-bce4-c7b262605968@mon.a 2024-09-06T14:30:15.915 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:30:15 smithi142 systemd[1]: Stopping Ceph mon.a for ce32462e-6c5a-11ef-bce4-c7b262605968... 2024-09-06T14:30:16.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:30:15 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mon-a[29954]: 2024-09-06T14:30:15.917+0000 7f33a2449640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-06T14:30:16.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:30:15 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mon-a[29954]: 2024-09-06T14:30:15.917+0000 7f33a2449640 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-09-06T14:30:16.209 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:30:16 smithi142 podman[80737]: 2024-09-06 14:30:16.100990063 +0000 UTC m=+0.293672306 container died e23b8f60e296b972502999b12c5cbedbb4c39b4bb7bc67fe0b8ad0da8551e445 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mon-a, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-athakkar-testing-2024-09-04-1837-8b38c33, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.build-date=20240731) 2024-09-06T14:30:16.709 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:30:16 smithi142 podman[80737]: 2024-09-06 14:30:16.609273769 +0000 UTC m=+0.801956011 container cleanup e23b8f60e296b972502999b12c5cbedbb4c39b4bb7bc67fe0b8ad0da8551e445 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mon-a, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-athakkar-testing-2024-09-04-1837-8b38c33, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-06T14:30:16.709 INFO:journalctl@ceph.mon.a.smithi142.stdout:Sep 06 14:30:16 smithi142 bash[80737]: ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mon-a 2024-09-06T14:30:17.036 DEBUG:teuthology.orchestra.run.smithi142:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ce32462e-6c5a-11ef-bce4-c7b262605968@mon.a.service' 2024-09-06T14:30:17.082 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-06T14:30:17.082 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-09-06T14:30:17.082 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-09-06T14:30:17.082 DEBUG:teuthology.orchestra.run.smithi142:> sudo systemctl stop ceph-ce32462e-6c5a-11ef-bce4-c7b262605968@mon.c 2024-09-06T14:30:17.428 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:30:17 smithi142 systemd[1]: Stopping Ceph mon.c for ce32462e-6c5a-11ef-bce4-c7b262605968... 2024-09-06T14:30:17.709 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:30:17 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mon-c[37368]: 2024-09-06T14:30:17.430+0000 7effc29b6640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-06T14:30:17.709 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:30:17 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mon-c[37368]: 2024-09-06T14:30:17.430+0000 7effc29b6640 -1 mon.c@2(peon) e3 *** Got Signal Terminated *** 2024-09-06T14:30:17.710 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:30:17 smithi142 podman[80881]: 2024-09-06 14:30:17.603984925 +0000 UTC m=+0.294996100 container died 991d8632ae3f036d4a3b7ad111243e1bda9dd55176b080f99983007d32f94452 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c86edd48571ab22373283621eb9feebce9df9c291d4164cfc4df1a09493e1f30, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mon-c, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-athakkar-testing-2024-09-04-1837-8b38c33, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-09-06T14:30:17.798 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 06 14:30:17 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-iscsi-iscsi-a[59853]: debug there is no tcmu-runner data available 2024-09-06T14:30:18.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:30:17 smithi142 podman[80881]: 2024-09-06 14:30:17.955307428 +0000 UTC m=+0.646318632 container cleanup 991d8632ae3f036d4a3b7ad111243e1bda9dd55176b080f99983007d32f94452 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c86edd48571ab22373283621eb9feebce9df9c291d4164cfc4df1a09493e1f30, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mon-c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, RELEASE=wip-athakkar-testing-2024-09-04-1837-8b38c33, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-09-06T14:30:18.209 INFO:journalctl@ceph.mon.c.smithi142.stdout:Sep 06 14:30:17 smithi142 bash[80881]: ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mon-c 2024-09-06T14:30:18.378 DEBUG:teuthology.orchestra.run.smithi142:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ce32462e-6c5a-11ef-bce4-c7b262605968@mon.c.service' 2024-09-06T14:30:18.412 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-06T14:30:18.412 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-09-06T14:30:18.412 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-09-06T14:30:18.412 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ceph-ce32462e-6c5a-11ef-bce4-c7b262605968@mon.b 2024-09-06T14:30:18.717 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:30:18 smithi162 systemd[1]: Stopping Ceph mon.b for ce32462e-6c5a-11ef-bce4-c7b262605968... 2024-09-06T14:30:19.048 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:30:18 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mon-b[33778]: 2024-09-06T14:30:18.720+0000 7fc16b0b0640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-06T14:30:19.048 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:30:18 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mon-b[33778]: 2024-09-06T14:30:18.720+0000 7fc16b0b0640 -1 mon.b@1(peon) e3 *** Got Signal Terminated *** 2024-09-06T14:30:19.048 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:30:18 smithi162 podman[65824]: 2024-09-06 14:30:18.923676809 +0000 UTC m=+0.318587050 container died b0a33a2233e77d93e1c1844b6c95da30b742fa1c107f78b66b2f5f8a000f4e0b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c86edd48571ab22373283621eb9feebce9df9c291d4164cfc4df1a09493e1f30, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mon-b, org.label-schema.build-date=20240731, RELEASE=wip-athakkar-testing-2024-09-04-1837-8b38c33, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-09-06T14:30:19.661 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:30:19 smithi162 podman[65824]: 2024-09-06 14:30:19.412365659 +0000 UTC m=+0.807275959 container cleanup b0a33a2233e77d93e1c1844b6c95da30b742fa1c107f78b66b2f5f8a000f4e0b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c86edd48571ab22373283621eb9feebce9df9c291d4164cfc4df1a09493e1f30, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mon-b, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-athakkar-testing-2024-09-04-1837-8b38c33, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731) 2024-09-06T14:30:19.661 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 06 14:30:19 smithi162 bash[65824]: ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mon-b 2024-09-06T14:30:19.786 DEBUG:teuthology.orchestra.run.smithi162:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ce32462e-6c5a-11ef-bce4-c7b262605968@mon.b.service' 2024-09-06T14:30:19.835 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-06T14:30:19.835 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-09-06T14:30:19.835 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-09-06T14:30:19.836 DEBUG:teuthology.orchestra.run.smithi142:> sudo systemctl stop ceph-ce32462e-6c5a-11ef-bce4-c7b262605968@mgr.y 2024-09-06T14:30:20.165 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:30:19 smithi142 systemd[1]: Stopping Ceph mgr.y for ce32462e-6c5a-11ef-bce4-c7b262605968... 2024-09-06T14:30:20.459 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:30:20 smithi142 podman[81004]: 2024-09-06 14:30:20.173657994 +0000 UTC m=+0.144765816 container died 3f6dd6c6ef80d91011b70aa7fbb1be61721ed05bebda3c56771f65f6d4465ae8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-athakkar-testing-2024-09-04-1837-8b38c33, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True) 2024-09-06T14:30:20.900 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:30:20 smithi142 podman[81004]: 2024-09-06 14:30:20.693569837 +0000 UTC m=+0.664677700 container cleanup 3f6dd6c6ef80d91011b70aa7fbb1be61721ed05bebda3c56771f65f6d4465ae8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8b38c33cecf4b0b7a9b3116eb9089e21e444ac73, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, RELEASE=wip-athakkar-testing-2024-09-04-1837-8b38c33, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-06T14:30:20.900 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Sep 06 14:30:20 smithi142 bash[81004]: ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-y 2024-09-06T14:30:21.117 DEBUG:teuthology.orchestra.run.smithi142:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ce32462e-6c5a-11ef-bce4-c7b262605968@mgr.y.service' 2024-09-06T14:30:21.152 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-06T14:30:21.152 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-09-06T14:30:21.152 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-09-06T14:30:21.152 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ceph-ce32462e-6c5a-11ef-bce4-c7b262605968@mgr.x 2024-09-06T14:30:21.462 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Sep 06 14:30:21 smithi162 systemd[1]: Stopping Ceph mgr.x for ce32462e-6c5a-11ef-bce4-c7b262605968... 2024-09-06T14:30:21.724 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Sep 06 14:30:21 smithi162 podman[65949]: 2024-09-06 14:30:21.465198535 +0000 UTC m=+0.117825415 container died 87bdbebbe6a4ddf59589404375c145948a1c2e531f0122bb3af28da2d62727d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c86edd48571ab22373283621eb9feebce9df9c291d4164cfc4df1a09493e1f30, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-x, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-athakkar-testing-2024-09-04-1837-8b38c33, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-06T14:30:22.048 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Sep 06 14:30:21 smithi162 podman[65949]: 2024-09-06 14:30:21.817291482 +0000 UTC m=+0.469918361 container cleanup 87bdbebbe6a4ddf59589404375c145948a1c2e531f0122bb3af28da2d62727d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c86edd48571ab22373283621eb9feebce9df9c291d4164cfc4df1a09493e1f30, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-x, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-athakkar-testing-2024-09-04-1837-8b38c33, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-09-06T14:30:22.048 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Sep 06 14:30:21 smithi162 bash[65949]: ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-mgr-x 2024-09-06T14:30:22.242 DEBUG:teuthology.orchestra.run.smithi162:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ce32462e-6c5a-11ef-bce4-c7b262605968@mgr.x.service' 2024-09-06T14:30:22.276 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-06T14:30:22.277 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-09-06T14:30:22.277 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-09-06T14:30:22.277 DEBUG:teuthology.orchestra.run.smithi142:> sudo systemctl stop ceph-ce32462e-6c5a-11ef-bce4-c7b262605968@osd.0 2024-09-06T14:30:22.594 INFO:journalctl@ceph.osd.0.smithi142.stdout:Sep 06 14:30:22 smithi142 systemd[1]: Stopping Ceph osd.0 for ce32462e-6c5a-11ef-bce4-c7b262605968... 2024-09-06T14:30:22.960 INFO:journalctl@ceph.osd.0.smithi142.stdout:Sep 06 14:30:22 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-0[41141]: 2024-09-06T14:30:22.597+0000 7f66d24dd640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-06T14:30:22.961 INFO:journalctl@ceph.osd.0.smithi142.stdout:Sep 06 14:30:22 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-0[41141]: 2024-09-06T14:30:22.597+0000 7f66d24dd640 -1 osd.0 64 *** Got signal Terminated *** 2024-09-06T14:30:22.961 INFO:journalctl@ceph.osd.0.smithi142.stdout:Sep 06 14:30:22 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-0[41141]: 2024-09-06T14:30:22.597+0000 7f66d24dd640 -1 osd.0 64 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-06T14:30:28.366 INFO:journalctl@ceph.osd.0.smithi142.stdout:Sep 06 14:30:28 smithi142 podman[81126]: 2024-09-06 14:30:28.077950994 +0000 UTC m=+5.601640029 container died b106e1c060ad905eb6027fb6d359f0da99780d55f54da7e68525c9392254f157 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c86edd48571ab22373283621eb9feebce9df9c291d4164cfc4df1a09493e1f30, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-0, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-athakkar-testing-2024-09-04-1837-8b38c33, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-09-06T14:30:28.709 INFO:journalctl@ceph.osd.0.smithi142.stdout:Sep 06 14:30:28 smithi142 podman[81126]: 2024-09-06 14:30:28.563867421 +0000 UTC m=+6.087556449 container cleanup b106e1c060ad905eb6027fb6d359f0da99780d55f54da7e68525c9392254f157 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c86edd48571ab22373283621eb9feebce9df9c291d4164cfc4df1a09493e1f30, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-0, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-athakkar-testing-2024-09-04-1837-8b38c33, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731) 2024-09-06T14:30:28.710 INFO:journalctl@ceph.osd.0.smithi142.stdout:Sep 06 14:30:28 smithi142 bash[81126]: ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-0 2024-09-06T14:30:29.180 INFO:journalctl@ceph.osd.0.smithi142.stdout:Sep 06 14:30:28 smithi142 podman[81141]: 2024-09-06 14:30:28.840539701 +0000 UTC m=+0.761128361 container remove b106e1c060ad905eb6027fb6d359f0da99780d55f54da7e68525c9392254f157 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c86edd48571ab22373283621eb9feebce9df9c291d4164cfc4df1a09493e1f30, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-athakkar-testing-2024-09-04-1837-8b38c33, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-09-06T14:30:29.441 INFO:journalctl@ceph.osd.0.smithi142.stdout:Sep 06 14:30:29 smithi142 podman[81235]: 2024-09-06 14:30:29.086557933 +0000 UTC m=+0.015737291 image pull 1c45aca0610a312233aa2613f7b416b2884ae14779f1297041485ab28f1c8110 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c86edd48571ab22373283621eb9feebce9df9c291d4164cfc4df1a09493e1f30 2024-09-06T14:30:29.441 INFO:journalctl@ceph.osd.0.smithi142.stdout:Sep 06 14:30:29 smithi142 podman[81235]: 2024-09-06 14:30:29.21642492 +0000 UTC m=+0.145604327 container create 8939abba337fbe6f1d919b5d4df129b2d95c5a84b94eab51a6d5bac4d94c6db8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c86edd48571ab22373283621eb9feebce9df9c291d4164cfc4df1a09493e1f30, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-0-deactivate, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, RELEASE=wip-athakkar-testing-2024-09-04-1837-8b38c33, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-09-06T14:30:29.441 INFO:journalctl@ceph.osd.0.smithi142.stdout:Sep 06 14:30:29 smithi142 podman[81235]: 2024-09-06 14:30:29.443893002 +0000 UTC m=+0.373072363 container init 8939abba337fbe6f1d919b5d4df129b2d95c5a84b94eab51a6d5bac4d94c6db8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c86edd48571ab22373283621eb9feebce9df9c291d4164cfc4df1a09493e1f30, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-athakkar-testing-2024-09-04-1837-8b38c33) 2024-09-06T14:30:29.710 INFO:journalctl@ceph.osd.0.smithi142.stdout:Sep 06 14:30:29 smithi142 podman[81235]: 2024-09-06 14:30:29.447486308 +0000 UTC m=+0.376665677 container start 8939abba337fbe6f1d919b5d4df129b2d95c5a84b94eab51a6d5bac4d94c6db8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c86edd48571ab22373283621eb9feebce9df9c291d4164cfc4df1a09493e1f30, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-athakkar-testing-2024-09-04-1837-8b38c33, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-09-06T14:30:29.710 INFO:journalctl@ceph.osd.0.smithi142.stdout:Sep 06 14:30:29 smithi142 podman[81235]: 2024-09-06 14:30:29.577059974 +0000 UTC m=+0.506239345 container attach 8939abba337fbe6f1d919b5d4df129b2d95c5a84b94eab51a6d5bac4d94c6db8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c86edd48571ab22373283621eb9feebce9df9c291d4164cfc4df1a09493e1f30, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-0-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, RELEASE=wip-athakkar-testing-2024-09-04-1837-8b38c33, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0) 2024-09-06T14:30:29.710 INFO:journalctl@ceph.osd.0.smithi142.stdout:Sep 06 14:30:29 smithi142 podman[81235]: 2024-09-06 14:30:29.639633673 +0000 UTC m=+0.568813039 container died 8939abba337fbe6f1d919b5d4df129b2d95c5a84b94eab51a6d5bac4d94c6db8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c86edd48571ab22373283621eb9feebce9df9c291d4164cfc4df1a09493e1f30, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-0-deactivate, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, RELEASE=wip-athakkar-testing-2024-09-04-1837-8b38c33, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-09-06T14:30:30.240 DEBUG:teuthology.orchestra.run.smithi142:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ce32462e-6c5a-11ef-bce4-c7b262605968@osd.0.service' 2024-09-06T14:30:30.281 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-06T14:30:30.281 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-09-06T14:30:30.281 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-09-06T14:30:30.281 DEBUG:teuthology.orchestra.run.smithi142:> sudo systemctl stop ceph-ce32462e-6c5a-11ef-bce4-c7b262605968@osd.1 2024-09-06T14:30:30.519 INFO:journalctl@ceph.osd.1.smithi142.stdout:Sep 06 14:30:30 smithi142 systemd[1]: Stopping Ceph osd.1 for ce32462e-6c5a-11ef-bce4-c7b262605968... 2024-09-06T14:30:30.959 INFO:journalctl@ceph.osd.1.smithi142.stdout:Sep 06 14:30:30 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-1[47186]: 2024-09-06T14:30:30.667+0000 7ff2d10fa640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-06T14:30:30.959 INFO:journalctl@ceph.osd.1.smithi142.stdout:Sep 06 14:30:30 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-1[47186]: 2024-09-06T14:30:30.667+0000 7ff2d10fa640 -1 osd.1 64 *** Got signal Terminated *** 2024-09-06T14:30:30.959 INFO:journalctl@ceph.osd.1.smithi142.stdout:Sep 06 14:30:30 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-1[47186]: 2024-09-06T14:30:30.667+0000 7ff2d10fa640 -1 osd.1 64 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-06T14:30:36.366 INFO:journalctl@ceph.osd.1.smithi142.stdout:Sep 06 14:30:36 smithi142 podman[81339]: 2024-09-06 14:30:36.062733 +0000 UTC m=+5.526873072 container died 827134f75148a40aff120289e289ed02cd9200439b32592c5114e612e212dbc7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c86edd48571ab22373283621eb9feebce9df9c291d4164cfc4df1a09493e1f30, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-1, ceph=True, org.label-schema.build-date=20240731, RELEASE=wip-athakkar-testing-2024-09-04-1837-8b38c33, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux ) 2024-09-06T14:30:36.709 INFO:journalctl@ceph.osd.1.smithi142.stdout:Sep 06 14:30:36 smithi142 podman[81339]: 2024-09-06 14:30:36.583804858 +0000 UTC m=+6.047944917 container cleanup 827134f75148a40aff120289e289ed02cd9200439b32592c5114e612e212dbc7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c86edd48571ab22373283621eb9feebce9df9c291d4164cfc4df1a09493e1f30, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-1, ceph=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-athakkar-testing-2024-09-04-1837-8b38c33, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-09-06T14:30:36.709 INFO:journalctl@ceph.osd.1.smithi142.stdout:Sep 06 14:30:36 smithi142 bash[81339]: ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-1 2024-09-06T14:30:37.209 INFO:journalctl@ceph.osd.1.smithi142.stdout:Sep 06 14:30:36 smithi142 podman[81365]: 2024-09-06 14:30:36.818617628 +0000 UTC m=+0.751939065 container remove 827134f75148a40aff120289e289ed02cd9200439b32592c5114e612e212dbc7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c86edd48571ab22373283621eb9feebce9df9c291d4164cfc4df1a09493e1f30, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-1, org.label-schema.schema-version=1.0, RELEASE=wip-athakkar-testing-2024-09-04-1837-8b38c33, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240731) 2024-09-06T14:30:37.488 INFO:journalctl@ceph.osd.1.smithi142.stdout:Sep 06 14:30:37 smithi142 podman[81458]: 2024-09-06 14:30:37.14329696 +0000 UTC m=+0.015883642 image pull 1c45aca0610a312233aa2613f7b416b2884ae14779f1297041485ab28f1c8110 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c86edd48571ab22373283621eb9feebce9df9c291d4164cfc4df1a09493e1f30 2024-09-06T14:30:37.489 INFO:journalctl@ceph.osd.1.smithi142.stdout:Sep 06 14:30:37 smithi142 podman[81458]: 2024-09-06 14:30:37.252592947 +0000 UTC m=+0.125179619 container create 6ea63c452758b02205ed9de78019c7fa6d7383f1d462b35cced967e00c57bbb8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c86edd48571ab22373283621eb9feebce9df9c291d4164cfc4df1a09493e1f30, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-1-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, RELEASE=wip-athakkar-testing-2024-09-04-1837-8b38c33, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731) 2024-09-06T14:30:37.489 INFO:journalctl@ceph.osd.1.smithi142.stdout:Sep 06 14:30:37 smithi142 podman[81458]: 2024-09-06 14:30:37.48857178 +0000 UTC m=+0.361158448 container init 6ea63c452758b02205ed9de78019c7fa6d7383f1d462b35cced967e00c57bbb8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c86edd48571ab22373283621eb9feebce9df9c291d4164cfc4df1a09493e1f30, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-1-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-athakkar-testing-2024-09-04-1837-8b38c33, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-09-06T14:30:37.489 INFO:journalctl@ceph.osd.1.smithi142.stdout:Sep 06 14:30:37 smithi142 podman[81458]: 2024-09-06 14:30:37.491818466 +0000 UTC m=+0.364405143 container start 6ea63c452758b02205ed9de78019c7fa6d7383f1d462b35cced967e00c57bbb8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c86edd48571ab22373283621eb9feebce9df9c291d4164cfc4df1a09493e1f30, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-1-deactivate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, RELEASE=wip-athakkar-testing-2024-09-04-1837-8b38c33, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-06T14:30:37.959 INFO:journalctl@ceph.osd.1.smithi142.stdout:Sep 06 14:30:37 smithi142 podman[81458]: 2024-09-06 14:30:37.605267218 +0000 UTC m=+0.477853889 container attach 6ea63c452758b02205ed9de78019c7fa6d7383f1d462b35cced967e00c57bbb8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c86edd48571ab22373283621eb9feebce9df9c291d4164cfc4df1a09493e1f30, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-1-deactivate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-athakkar-testing-2024-09-04-1837-8b38c33, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2) 2024-09-06T14:30:37.959 INFO:journalctl@ceph.osd.1.smithi142.stdout:Sep 06 14:30:37 smithi142 podman[81458]: 2024-09-06 14:30:37.681481759 +0000 UTC m=+0.554068438 container died 6ea63c452758b02205ed9de78019c7fa6d7383f1d462b35cced967e00c57bbb8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c86edd48571ab22373283621eb9feebce9df9c291d4164cfc4df1a09493e1f30, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-1-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, RELEASE=wip-athakkar-testing-2024-09-04-1837-8b38c33) 2024-09-06T14:30:38.357 DEBUG:teuthology.orchestra.run.smithi142:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ce32462e-6c5a-11ef-bce4-c7b262605968@osd.1.service' 2024-09-06T14:30:38.396 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-06T14:30:38.397 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-09-06T14:30:38.397 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-09-06T14:30:38.397 DEBUG:teuthology.orchestra.run.smithi142:> sudo systemctl stop ceph-ce32462e-6c5a-11ef-bce4-c7b262605968@osd.2 2024-09-06T14:30:38.710 INFO:journalctl@ceph.osd.2.smithi142.stdout:Sep 06 14:30:38 smithi142 systemd[1]: Stopping Ceph osd.2 for ce32462e-6c5a-11ef-bce4-c7b262605968... 2024-09-06T14:30:39.209 INFO:journalctl@ceph.osd.2.smithi142.stdout:Sep 06 14:30:38 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-2[52762]: 2024-09-06T14:30:38.753+0000 7f2d01211640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-06T14:30:39.210 INFO:journalctl@ceph.osd.2.smithi142.stdout:Sep 06 14:30:38 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-2[52762]: 2024-09-06T14:30:38.753+0000 7f2d01211640 -1 osd.2 64 *** Got signal Terminated *** 2024-09-06T14:30:39.210 INFO:journalctl@ceph.osd.2.smithi142.stdout:Sep 06 14:30:38 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-2[52762]: 2024-09-06T14:30:38.753+0000 7f2d01211640 -1 osd.2 64 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-06T14:30:44.501 INFO:journalctl@ceph.osd.2.smithi142.stdout:Sep 06 14:30:44 smithi142 podman[81560]: 2024-09-06 14:30:44.218439691 +0000 UTC m=+5.594600767 container died 66393355d7ab58abf85552780ab8ea4f37aa1d1e748c281c6b18f0a5b5aaa4af (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c86edd48571ab22373283621eb9feebce9df9c291d4164cfc4df1a09493e1f30, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-athakkar-testing-2024-09-04-1837-8b38c33, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-06T14:30:44.959 INFO:journalctl@ceph.osd.2.smithi142.stdout:Sep 06 14:30:44 smithi142 podman[81560]: 2024-09-06 14:30:44.694131584 +0000 UTC m=+6.070292702 container cleanup 66393355d7ab58abf85552780ab8ea4f37aa1d1e748c281c6b18f0a5b5aaa4af (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c86edd48571ab22373283621eb9feebce9df9c291d4164cfc4df1a09493e1f30, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-2, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-athakkar-testing-2024-09-04-1837-8b38c33, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-09-06T14:30:44.959 INFO:journalctl@ceph.osd.2.smithi142.stdout:Sep 06 14:30:44 smithi142 bash[81560]: ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-2 2024-09-06T14:30:45.209 INFO:journalctl@ceph.osd.2.smithi142.stdout:Sep 06 14:30:44 smithi142 podman[81575]: 2024-09-06 14:30:44.961968971 +0000 UTC m=+0.744983486 container remove 66393355d7ab58abf85552780ab8ea4f37aa1d1e748c281c6b18f0a5b5aaa4af (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c86edd48571ab22373283621eb9feebce9df9c291d4164cfc4df1a09493e1f30, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, RELEASE=wip-athakkar-testing-2024-09-04-1837-8b38c33, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0) 2024-09-06T14:30:45.554 INFO:journalctl@ceph.osd.2.smithi142.stdout:Sep 06 14:30:45 smithi142 podman[81665]: 2024-09-06 14:30:45.209309611 +0000 UTC m=+0.015707153 image pull 1c45aca0610a312233aa2613f7b416b2884ae14779f1297041485ab28f1c8110 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c86edd48571ab22373283621eb9feebce9df9c291d4164cfc4df1a09493e1f30 2024-09-06T14:30:45.554 INFO:journalctl@ceph.osd.2.smithi142.stdout:Sep 06 14:30:45 smithi142 podman[81665]: 2024-09-06 14:30:45.321184832 +0000 UTC m=+0.127582366 container create cee89c0f304b44d06e3131a58691d422ac735e9f1cc9bafe7a39a2ea519c547c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c86edd48571ab22373283621eb9feebce9df9c291d4164cfc4df1a09493e1f30, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-2-deactivate, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, RELEASE=wip-athakkar-testing-2024-09-04-1837-8b38c33, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0) 2024-09-06T14:30:45.554 INFO:journalctl@ceph.osd.2.smithi142.stdout:Sep 06 14:30:45 smithi142 podman[81665]: 2024-09-06 14:30:45.556948262 +0000 UTC m=+0.363345796 container init cee89c0f304b44d06e3131a58691d422ac735e9f1cc9bafe7a39a2ea519c547c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c86edd48571ab22373283621eb9feebce9df9c291d4164cfc4df1a09493e1f30, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-2-deactivate, GIT_CLEAN=True, RELEASE=wip-athakkar-testing-2024-09-04-1837-8b38c33, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2) 2024-09-06T14:30:45.959 INFO:journalctl@ceph.osd.2.smithi142.stdout:Sep 06 14:30:45 smithi142 podman[81665]: 2024-09-06 14:30:45.560336277 +0000 UTC m=+0.366733814 container start cee89c0f304b44d06e3131a58691d422ac735e9f1cc9bafe7a39a2ea519c547c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c86edd48571ab22373283621eb9feebce9df9c291d4164cfc4df1a09493e1f30, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-2-deactivate, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, RELEASE=wip-athakkar-testing-2024-09-04-1837-8b38c33, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-06T14:30:45.960 INFO:journalctl@ceph.osd.2.smithi142.stdout:Sep 06 14:30:45 smithi142 podman[81665]: 2024-09-06 14:30:45.665294086 +0000 UTC m=+0.471691629 container attach cee89c0f304b44d06e3131a58691d422ac735e9f1cc9bafe7a39a2ea519c547c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c86edd48571ab22373283621eb9feebce9df9c291d4164cfc4df1a09493e1f30, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-2-deactivate, ceph=True, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, RELEASE=wip-athakkar-testing-2024-09-04-1837-8b38c33, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-09-06T14:30:45.960 INFO:journalctl@ceph.osd.2.smithi142.stdout:Sep 06 14:30:45 smithi142 podman[81665]: 2024-09-06 14:30:45.749281003 +0000 UTC m=+0.555678550 container died cee89c0f304b44d06e3131a58691d422ac735e9f1cc9bafe7a39a2ea519c547c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c86edd48571ab22373283621eb9feebce9df9c291d4164cfc4df1a09493e1f30, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-2-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-athakkar-testing-2024-09-04-1837-8b38c33, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2) 2024-09-06T14:30:46.402 DEBUG:teuthology.orchestra.run.smithi142:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ce32462e-6c5a-11ef-bce4-c7b262605968@osd.2.service' 2024-09-06T14:30:46.441 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-06T14:30:46.441 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-09-06T14:30:46.441 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-09-06T14:30:46.441 DEBUG:teuthology.orchestra.run.smithi142:> sudo systemctl stop ceph-ce32462e-6c5a-11ef-bce4-c7b262605968@osd.3 2024-09-06T14:30:46.709 INFO:journalctl@ceph.osd.3.smithi142.stdout:Sep 06 14:30:46 smithi142 systemd[1]: Stopping Ceph osd.3 for ce32462e-6c5a-11ef-bce4-c7b262605968... 2024-09-06T14:30:47.209 INFO:journalctl@ceph.osd.3.smithi142.stdout:Sep 06 14:30:46 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-3[58430]: 2024-09-06T14:30:46.779+0000 7fb0b2d95640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-06T14:30:47.209 INFO:journalctl@ceph.osd.3.smithi142.stdout:Sep 06 14:30:46 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-3[58430]: 2024-09-06T14:30:46.779+0000 7fb0b2d95640 -1 osd.3 64 *** Got signal Terminated *** 2024-09-06T14:30:47.209 INFO:journalctl@ceph.osd.3.smithi142.stdout:Sep 06 14:30:46 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-3[58430]: 2024-09-06T14:30:46.779+0000 7fb0b2d95640 -1 osd.3 64 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-06T14:30:49.719 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 06 14:30:49 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-4[38539]: 2024-09-06T14:30:49.354+0000 7f36a90ce640 -1 osd.4 64 heartbeat_check: no reply from 172.21.15.142:6803 osd.0 since back 2024-09-06T14:30:24.605306+0000 front 2024-09-06T14:30:24.605599+0000 (oldest deadline 2024-09-06T14:30:48.705372+0000) 2024-09-06T14:30:49.960 INFO:journalctl@ceph.osd.3.smithi142.stdout:Sep 06 14:30:49 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-3[58430]: 2024-09-06T14:30:49.650+0000 7fb0ab3a2640 -1 osd.3 64 heartbeat_check: no reply from 172.21.15.142:6803 osd.0 since back 2024-09-06T14:30:24.026142+0000 front 2024-09-06T14:30:24.026215+0000 (oldest deadline 2024-09-06T14:30:49.326259+0000) 2024-09-06T14:30:50.298 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:30:50 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7[55156]: 2024-09-06T14:30:50.018+0000 7fa8a1380640 -1 osd.7 64 heartbeat_check: no reply from 172.21.15.142:6803 osd.0 since back 2024-09-06T14:30:25.832410+0000 front 2024-09-06T14:30:25.832610+0000 (oldest deadline 2024-09-06T14:30:49.332618+0000) 2024-09-06T14:30:50.798 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 06 14:30:50 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-4[38539]: 2024-09-06T14:30:50.388+0000 7f36a90ce640 -1 osd.4 64 heartbeat_check: no reply from 172.21.15.142:6803 osd.0 since back 2024-09-06T14:30:24.605306+0000 front 2024-09-06T14:30:24.605599+0000 (oldest deadline 2024-09-06T14:30:48.705372+0000) 2024-09-06T14:30:50.798 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 06 14:30:50 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-5[44099]: 2024-09-06T14:30:50.623+0000 7fbac27a0640 -1 osd.5 64 heartbeat_check: no reply from 172.21.15.142:6803 osd.0 since back 2024-09-06T14:30:24.533858+0000 front 2024-09-06T14:30:24.534023+0000 (oldest deadline 2024-09-06T14:30:50.433941+0000) 2024-09-06T14:30:50.959 INFO:journalctl@ceph.osd.3.smithi142.stdout:Sep 06 14:30:50 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-3[58430]: 2024-09-06T14:30:50.669+0000 7fb0ab3a2640 -1 osd.3 64 heartbeat_check: no reply from 172.21.15.142:6803 osd.0 since back 2024-09-06T14:30:24.026142+0000 front 2024-09-06T14:30:24.026215+0000 (oldest deadline 2024-09-06T14:30:49.326259+0000) 2024-09-06T14:30:51.297 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:30:50 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7[55156]: 2024-09-06T14:30:50.977+0000 7fa8a1380640 -1 osd.7 64 heartbeat_check: no reply from 172.21.15.142:6803 osd.0 since back 2024-09-06T14:30:25.832410+0000 front 2024-09-06T14:30:25.832610+0000 (oldest deadline 2024-09-06T14:30:49.332618+0000) 2024-09-06T14:30:51.626 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 06 14:30:51 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-6[49599]: 2024-09-06T14:30:51.302+0000 7fef2ba64640 -1 osd.6 64 heartbeat_check: no reply from 172.21.15.142:6803 osd.0 since back 2024-09-06T14:30:25.822692+0000 front 2024-09-06T14:30:25.822628+0000 (oldest deadline 2024-09-06T14:30:51.122413+0000) 2024-09-06T14:30:51.626 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 06 14:30:51 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-4[38539]: 2024-09-06T14:30:51.374+0000 7f36a90ce640 -1 osd.4 64 heartbeat_check: no reply from 172.21.15.142:6803 osd.0 since back 2024-09-06T14:30:24.605306+0000 front 2024-09-06T14:30:24.605599+0000 (oldest deadline 2024-09-06T14:30:48.705372+0000) 2024-09-06T14:30:51.928 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 06 14:30:51 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-5[44099]: 2024-09-06T14:30:51.628+0000 7fbac27a0640 -1 osd.5 64 heartbeat_check: no reply from 172.21.15.142:6803 osd.0 since back 2024-09-06T14:30:24.533858+0000 front 2024-09-06T14:30:24.534023+0000 (oldest deadline 2024-09-06T14:30:50.433941+0000) 2024-09-06T14:30:51.959 INFO:journalctl@ceph.osd.3.smithi142.stdout:Sep 06 14:30:51 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-3[58430]: 2024-09-06T14:30:51.653+0000 7fb0ab3a2640 -1 osd.3 64 heartbeat_check: no reply from 172.21.15.142:6803 osd.0 since back 2024-09-06T14:30:24.026142+0000 front 2024-09-06T14:30:24.026215+0000 (oldest deadline 2024-09-06T14:30:49.326259+0000) 2024-09-06T14:30:52.595 INFO:journalctl@ceph.osd.3.smithi142.stdout:Sep 06 14:30:52 smithi142 podman[81767]: 2024-09-06 14:30:52.275381869 +0000 UTC m=+5.609594113 container died ee5e7ca56b9ad2d63b7ea0b7198a8d82c4975d5356e48814973926e60b505af8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c86edd48571ab22373283621eb9feebce9df9c291d4164cfc4df1a09493e1f30, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-3, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-athakkar-testing-2024-09-04-1837-8b38c33, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2) 2024-09-06T14:30:52.626 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 06 14:30:52 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-6[49599]: 2024-09-06T14:30:52.321+0000 7fef2ba64640 -1 osd.6 64 heartbeat_check: no reply from 172.21.15.142:6803 osd.0 since back 2024-09-06T14:30:25.822692+0000 front 2024-09-06T14:30:25.822628+0000 (oldest deadline 2024-09-06T14:30:51.122413+0000) 2024-09-06T14:30:52.626 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:30:51 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7[55156]: 2024-09-06T14:30:51.931+0000 7fa8a1380640 -1 osd.7 64 heartbeat_check: no reply from 172.21.15.142:6803 osd.0 since back 2024-09-06T14:30:25.832410+0000 front 2024-09-06T14:30:25.832610+0000 (oldest deadline 2024-09-06T14:30:49.332618+0000) 2024-09-06T14:30:52.626 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 06 14:30:52 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-4[38539]: 2024-09-06T14:30:52.399+0000 7f36a90ce640 -1 osd.4 64 heartbeat_check: no reply from 172.21.15.142:6803 osd.0 since back 2024-09-06T14:30:24.605306+0000 front 2024-09-06T14:30:24.605599+0000 (oldest deadline 2024-09-06T14:30:48.705372+0000) 2024-09-06T14:30:52.926 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 06 14:30:52 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-5[44099]: 2024-09-06T14:30:52.628+0000 7fbac27a0640 -1 osd.5 64 heartbeat_check: no reply from 172.21.15.142:6803 osd.0 since back 2024-09-06T14:30:24.533858+0000 front 2024-09-06T14:30:24.534023+0000 (oldest deadline 2024-09-06T14:30:50.433941+0000) 2024-09-06T14:30:52.959 INFO:journalctl@ceph.osd.3.smithi142.stdout:Sep 06 14:30:52 smithi142 podman[81767]: 2024-09-06 14:30:52.78004305 +0000 UTC m=+6.114255294 container cleanup ee5e7ca56b9ad2d63b7ea0b7198a8d82c4975d5356e48814973926e60b505af8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c86edd48571ab22373283621eb9feebce9df9c291d4164cfc4df1a09493e1f30, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-3, org.label-schema.build-date=20240731, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-athakkar-testing-2024-09-04-1837-8b38c33, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-09-06T14:30:52.960 INFO:journalctl@ceph.osd.3.smithi142.stdout:Sep 06 14:30:52 smithi142 bash[81767]: ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-3 2024-09-06T14:30:53.297 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:30:52 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7[55156]: 2024-09-06T14:30:52.928+0000 7fa8a1380640 -1 osd.7 64 heartbeat_check: no reply from 172.21.15.142:6803 osd.0 since back 2024-09-06T14:30:25.832410+0000 front 2024-09-06T14:30:25.832610+0000 (oldest deadline 2024-09-06T14:30:49.332618+0000) 2024-09-06T14:30:53.387 INFO:journalctl@ceph.osd.3.smithi142.stdout:Sep 06 14:30:53 smithi142 podman[81783]: 2024-09-06 14:30:53.039544989 +0000 UTC m=+0.754077444 container remove ee5e7ca56b9ad2d63b7ea0b7198a8d82c4975d5356e48814973926e60b505af8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c86edd48571ab22373283621eb9feebce9df9c291d4164cfc4df1a09493e1f30, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-3, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240731, RELEASE=wip-athakkar-testing-2024-09-04-1837-8b38c33) 2024-09-06T14:30:53.709 INFO:journalctl@ceph.osd.3.smithi142.stdout:Sep 06 14:30:53 smithi142 podman[81875]: 2024-09-06 14:30:53.292673251 +0000 UTC m=+0.015560708 image pull 1c45aca0610a312233aa2613f7b416b2884ae14779f1297041485ab28f1c8110 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c86edd48571ab22373283621eb9feebce9df9c291d4164cfc4df1a09493e1f30 2024-09-06T14:30:53.709 INFO:journalctl@ceph.osd.3.smithi142.stdout:Sep 06 14:30:53 smithi142 podman[81875]: 2024-09-06 14:30:53.406885167 +0000 UTC m=+0.129772610 container create 99c22f53185373605efdd1fe63672e1acef79e5a8cf80394e0c22d0e3f0bf47c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c86edd48571ab22373283621eb9feebce9df9c291d4164cfc4df1a09493e1f30, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-3-deactivate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, RELEASE=wip-athakkar-testing-2024-09-04-1837-8b38c33, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-09-06T14:30:53.710 INFO:journalctl@ceph.osd.3.smithi142.stdout:Sep 06 14:30:53 smithi142 podman[81875]: 2024-09-06 14:30:53.633984864 +0000 UTC m=+0.356872313 container init 99c22f53185373605efdd1fe63672e1acef79e5a8cf80394e0c22d0e3f0bf47c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c86edd48571ab22373283621eb9feebce9df9c291d4164cfc4df1a09493e1f30, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-3-deactivate, ceph=True, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-athakkar-testing-2024-09-04-1837-8b38c33, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-06T14:30:53.710 INFO:journalctl@ceph.osd.3.smithi142.stdout:Sep 06 14:30:53 smithi142 podman[81875]: 2024-09-06 14:30:53.637393641 +0000 UTC m=+0.360281083 container start 99c22f53185373605efdd1fe63672e1acef79e5a8cf80394e0c22d0e3f0bf47c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c86edd48571ab22373283621eb9feebce9df9c291d4164cfc4df1a09493e1f30, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-3-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.37.2, RELEASE=wip-athakkar-testing-2024-09-04-1837-8b38c33, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_CLEAN=True) 2024-09-06T14:30:53.798 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 06 14:30:53 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-4[38539]: 2024-09-06T14:30:53.438+0000 7f36a90ce640 -1 osd.4 64 heartbeat_check: no reply from 172.21.15.142:6803 osd.0 since back 2024-09-06T14:30:24.605306+0000 front 2024-09-06T14:30:24.605599+0000 (oldest deadline 2024-09-06T14:30:48.705372+0000) 2024-09-06T14:30:53.798 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 06 14:30:53 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-5[44099]: 2024-09-06T14:30:53.596+0000 7fbac27a0640 -1 osd.5 64 heartbeat_check: no reply from 172.21.15.142:6803 osd.0 since back 2024-09-06T14:30:24.533858+0000 front 2024-09-06T14:30:24.534023+0000 (oldest deadline 2024-09-06T14:30:50.433941+0000) 2024-09-06T14:30:53.798 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 06 14:30:53 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-6[49599]: 2024-09-06T14:30:53.368+0000 7fef2ba64640 -1 osd.6 64 heartbeat_check: no reply from 172.21.15.142:6803 osd.0 since back 2024-09-06T14:30:25.822692+0000 front 2024-09-06T14:30:25.822628+0000 (oldest deadline 2024-09-06T14:30:51.122413+0000) 2024-09-06T14:30:54.175 INFO:journalctl@ceph.osd.3.smithi142.stdout:Sep 06 14:30:53 smithi142 podman[81875]: 2024-09-06 14:30:53.717301359 +0000 UTC m=+0.440188807 container attach 99c22f53185373605efdd1fe63672e1acef79e5a8cf80394e0c22d0e3f0bf47c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c86edd48571ab22373283621eb9feebce9df9c291d4164cfc4df1a09493e1f30, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-3-deactivate, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-athakkar-testing-2024-09-04-1837-8b38c33) 2024-09-06T14:30:54.175 INFO:journalctl@ceph.osd.3.smithi142.stdout:Sep 06 14:30:53 smithi142 podman[81875]: 2024-09-06 14:30:53.829403362 +0000 UTC m=+0.552290806 container died 99c22f53185373605efdd1fe63672e1acef79e5a8cf80394e0c22d0e3f0bf47c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c86edd48571ab22373283621eb9feebce9df9c291d4164cfc4df1a09493e1f30, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-3-deactivate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, RELEASE=wip-athakkar-testing-2024-09-04-1837-8b38c33, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-09-06T14:30:54.298 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:30:53 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7[55156]: 2024-09-06T14:30:53.963+0000 7fa8a1380640 -1 osd.7 64 heartbeat_check: no reply from 172.21.15.142:6803 osd.0 since back 2024-09-06T14:30:25.832410+0000 front 2024-09-06T14:30:25.832610+0000 (oldest deadline 2024-09-06T14:30:49.332618+0000) 2024-09-06T14:30:54.560 DEBUG:teuthology.orchestra.run.smithi142:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ce32462e-6c5a-11ef-bce4-c7b262605968@osd.3.service' 2024-09-06T14:30:54.599 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-06T14:30:54.599 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-09-06T14:30:54.600 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-09-06T14:30:54.600 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ceph-ce32462e-6c5a-11ef-bce4-c7b262605968@osd.4 2024-09-06T14:30:54.651 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 06 14:30:54 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-4[38539]: 2024-09-06T14:30:54.409+0000 7f36a90ce640 -1 osd.4 64 heartbeat_check: no reply from 172.21.15.142:6803 osd.0 since back 2024-09-06T14:30:24.605306+0000 front 2024-09-06T14:30:24.605599+0000 (oldest deadline 2024-09-06T14:30:48.705372+0000) 2024-09-06T14:30:54.652 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 06 14:30:54 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-5[44099]: 2024-09-06T14:30:54.589+0000 7fbac27a0640 -1 osd.5 64 heartbeat_check: no reply from 172.21.15.142:6803 osd.0 since back 2024-09-06T14:30:24.533858+0000 front 2024-09-06T14:30:24.534023+0000 (oldest deadline 2024-09-06T14:30:50.433941+0000) 2024-09-06T14:30:54.652 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 06 14:30:54 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-6[49599]: 2024-09-06T14:30:54.394+0000 7fef2ba64640 -1 osd.6 64 heartbeat_check: no reply from 172.21.15.142:6803 osd.0 since back 2024-09-06T14:30:25.822692+0000 front 2024-09-06T14:30:25.822628+0000 (oldest deadline 2024-09-06T14:30:51.122413+0000) 2024-09-06T14:30:54.911 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 06 14:30:54 smithi162 systemd[1]: Stopping Ceph osd.4 for ce32462e-6c5a-11ef-bce4-c7b262605968... 2024-09-06T14:30:55.298 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:30:54 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7[55156]: 2024-09-06T14:30:54.913+0000 7fa8a1380640 -1 osd.7 64 heartbeat_check: no reply from 172.21.15.142:6803 osd.0 since back 2024-09-06T14:30:25.832410+0000 front 2024-09-06T14:30:25.832610+0000 (oldest deadline 2024-09-06T14:30:49.332618+0000) 2024-09-06T14:30:55.298 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 06 14:30:54 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-4[38539]: 2024-09-06T14:30:54.924+0000 7f36acab9640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-06T14:30:55.298 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 06 14:30:54 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-4[38539]: 2024-09-06T14:30:54.924+0000 7f36acab9640 -1 osd.4 64 *** Got signal Terminated *** 2024-09-06T14:30:55.298 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 06 14:30:54 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-4[38539]: 2024-09-06T14:30:54.924+0000 7f36acab9640 -1 osd.4 64 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-06T14:30:55.627 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 06 14:30:55 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-4[38539]: 2024-09-06T14:30:55.389+0000 7f36a90ce640 -1 osd.4 64 heartbeat_check: no reply from 172.21.15.142:6803 osd.0 since back 2024-09-06T14:30:24.605306+0000 front 2024-09-06T14:30:24.605599+0000 (oldest deadline 2024-09-06T14:30:48.705372+0000) 2024-09-06T14:30:55.627 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 06 14:30:55 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-6[49599]: 2024-09-06T14:30:55.362+0000 7fef2ba64640 -1 osd.6 64 heartbeat_check: no reply from 172.21.15.142:6803 osd.0 since back 2024-09-06T14:30:25.822692+0000 front 2024-09-06T14:30:25.822628+0000 (oldest deadline 2024-09-06T14:30:51.122413+0000) 2024-09-06T14:30:55.893 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 06 14:30:55 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-5[44099]: 2024-09-06T14:30:55.628+0000 7fbac27a0640 -1 osd.5 64 heartbeat_check: no reply from 172.21.15.142:6803 osd.0 since back 2024-09-06T14:30:24.533858+0000 front 2024-09-06T14:30:24.534023+0000 (oldest deadline 2024-09-06T14:30:50.433941+0000) 2024-09-06T14:30:56.298 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:30:55 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7[55156]: 2024-09-06T14:30:55.896+0000 7fa8a1380640 -1 osd.7 64 heartbeat_check: no reply from 172.21.15.142:6803 osd.0 since back 2024-09-06T14:30:25.832410+0000 front 2024-09-06T14:30:25.832610+0000 (oldest deadline 2024-09-06T14:30:49.332618+0000) 2024-09-06T14:30:56.676 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 06 14:30:56 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-6[49599]: 2024-09-06T14:30:56.407+0000 7fef2ba64640 -1 osd.6 64 heartbeat_check: no reply from 172.21.15.142:6803 osd.0 since back 2024-09-06T14:30:25.822692+0000 front 2024-09-06T14:30:25.822628+0000 (oldest deadline 2024-09-06T14:30:51.122413+0000) 2024-09-06T14:30:56.676 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 06 14:30:56 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-4[38539]: 2024-09-06T14:30:56.390+0000 7f36a90ce640 -1 osd.4 64 heartbeat_check: no reply from 172.21.15.142:6803 osd.0 since back 2024-09-06T14:30:24.605306+0000 front 2024-09-06T14:30:24.605599+0000 (oldest deadline 2024-09-06T14:30:48.705372+0000) 2024-09-06T14:30:56.939 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 06 14:30:56 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-5[44099]: 2024-09-06T14:30:56.678+0000 7fbac27a0640 -1 osd.5 64 heartbeat_check: no reply from 172.21.15.142:6803 osd.0 since back 2024-09-06T14:30:24.533858+0000 front 2024-09-06T14:30:24.534023+0000 (oldest deadline 2024-09-06T14:30:50.433941+0000) 2024-09-06T14:30:57.298 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:30:56 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7[55156]: 2024-09-06T14:30:56.941+0000 7fa8a1380640 -1 osd.7 64 heartbeat_check: no reply from 172.21.15.142:6803 osd.0 since back 2024-09-06T14:30:25.832410+0000 front 2024-09-06T14:30:25.832610+0000 (oldest deadline 2024-09-06T14:30:49.332618+0000) 2024-09-06T14:30:57.298 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:30:56 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7[55156]: 2024-09-06T14:30:56.941+0000 7fa8a1380640 -1 osd.7 64 heartbeat_check: no reply from 172.21.15.142:6807 osd.1 since back 2024-09-06T14:30:31.633604+0000 front 2024-09-06T14:30:31.633572+0000 (oldest deadline 2024-09-06T14:30:56.933451+0000) 2024-09-06T14:30:57.798 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 06 14:30:57 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-4[38539]: 2024-09-06T14:30:57.429+0000 7f36a90ce640 -1 osd.4 64 heartbeat_check: no reply from 172.21.15.142:6803 osd.0 since back 2024-09-06T14:30:24.605306+0000 front 2024-09-06T14:30:24.605599+0000 (oldest deadline 2024-09-06T14:30:48.705372+0000) 2024-09-06T14:30:57.798 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 06 14:30:57 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-5[44099]: 2024-09-06T14:30:57.639+0000 7fbac27a0640 -1 osd.5 64 heartbeat_check: no reply from 172.21.15.142:6803 osd.0 since back 2024-09-06T14:30:24.533858+0000 front 2024-09-06T14:30:24.534023+0000 (oldest deadline 2024-09-06T14:30:50.433941+0000) 2024-09-06T14:30:57.798 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 06 14:30:57 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-5[44099]: 2024-09-06T14:30:57.639+0000 7fbac27a0640 -1 osd.5 64 heartbeat_check: no reply from 172.21.15.142:6807 osd.1 since back 2024-09-06T14:30:33.334947+0000 front 2024-09-06T14:30:33.335179+0000 (oldest deadline 2024-09-06T14:30:56.835064+0000) 2024-09-06T14:30:57.798 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 06 14:30:57 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-6[49599]: 2024-09-06T14:30:57.403+0000 7fef2ba64640 -1 osd.6 64 heartbeat_check: no reply from 172.21.15.142:6803 osd.0 since back 2024-09-06T14:30:25.822692+0000 front 2024-09-06T14:30:25.822628+0000 (oldest deadline 2024-09-06T14:30:51.122413+0000) 2024-09-06T14:30:57.798 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 06 14:30:57 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-6[49599]: 2024-09-06T14:30:57.403+0000 7fef2ba64640 -1 osd.6 64 heartbeat_check: no reply from 172.21.15.142:6807 osd.1 since back 2024-09-06T14:30:31.122845+0000 front 2024-09-06T14:30:31.122767+0000 (oldest deadline 2024-09-06T14:30:57.022798+0000) 2024-09-06T14:30:58.298 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:30:57 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7[55156]: 2024-09-06T14:30:57.894+0000 7fa8a1380640 -1 osd.7 64 heartbeat_check: no reply from 172.21.15.142:6803 osd.0 since back 2024-09-06T14:30:25.832410+0000 front 2024-09-06T14:30:25.832610+0000 (oldest deadline 2024-09-06T14:30:49.332618+0000) 2024-09-06T14:30:58.298 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:30:57 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7[55156]: 2024-09-06T14:30:57.894+0000 7fa8a1380640 -1 osd.7 64 heartbeat_check: no reply from 172.21.15.142:6807 osd.1 since back 2024-09-06T14:30:31.633604+0000 front 2024-09-06T14:30:31.633572+0000 (oldest deadline 2024-09-06T14:30:56.933451+0000) 2024-09-06T14:30:58.798 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 06 14:30:58 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-4[38539]: 2024-09-06T14:30:58.410+0000 7f36a90ce640 -1 osd.4 64 heartbeat_check: no reply from 172.21.15.142:6803 osd.0 since back 2024-09-06T14:30:24.605306+0000 front 2024-09-06T14:30:24.605599+0000 (oldest deadline 2024-09-06T14:30:48.705372+0000) 2024-09-06T14:30:58.798 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 06 14:30:58 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-5[44099]: 2024-09-06T14:30:58.629+0000 7fbac27a0640 -1 osd.5 64 heartbeat_check: no reply from 172.21.15.142:6803 osd.0 since back 2024-09-06T14:30:24.533858+0000 front 2024-09-06T14:30:24.534023+0000 (oldest deadline 2024-09-06T14:30:50.433941+0000) 2024-09-06T14:30:58.798 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 06 14:30:58 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-5[44099]: 2024-09-06T14:30:58.629+0000 7fbac27a0640 -1 osd.5 64 heartbeat_check: no reply from 172.21.15.142:6807 osd.1 since back 2024-09-06T14:30:33.334947+0000 front 2024-09-06T14:30:33.335179+0000 (oldest deadline 2024-09-06T14:30:56.835064+0000) 2024-09-06T14:30:58.798 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 06 14:30:58 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-6[49599]: 2024-09-06T14:30:58.448+0000 7fef2ba64640 -1 osd.6 64 heartbeat_check: no reply from 172.21.15.142:6803 osd.0 since back 2024-09-06T14:30:25.822692+0000 front 2024-09-06T14:30:25.822628+0000 (oldest deadline 2024-09-06T14:30:51.122413+0000) 2024-09-06T14:30:58.798 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 06 14:30:58 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-6[49599]: 2024-09-06T14:30:58.448+0000 7fef2ba64640 -1 osd.6 64 heartbeat_check: no reply from 172.21.15.142:6807 osd.1 since back 2024-09-06T14:30:31.122845+0000 front 2024-09-06T14:30:31.122767+0000 (oldest deadline 2024-09-06T14:30:57.022798+0000) 2024-09-06T14:30:59.298 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:30:58 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7[55156]: 2024-09-06T14:30:58.874+0000 7fa8a1380640 -1 osd.7 64 heartbeat_check: no reply from 172.21.15.142:6803 osd.0 since back 2024-09-06T14:30:25.832410+0000 front 2024-09-06T14:30:25.832610+0000 (oldest deadline 2024-09-06T14:30:49.332618+0000) 2024-09-06T14:30:59.298 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:30:58 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7[55156]: 2024-09-06T14:30:58.874+0000 7fa8a1380640 -1 osd.7 64 heartbeat_check: no reply from 172.21.15.142:6807 osd.1 since back 2024-09-06T14:30:31.633604+0000 front 2024-09-06T14:30:31.633572+0000 (oldest deadline 2024-09-06T14:30:56.933451+0000) 2024-09-06T14:30:59.638 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 06 14:30:59 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-4[38539]: 2024-09-06T14:30:59.367+0000 7f36a90ce640 -1 osd.4 64 heartbeat_check: no reply from 172.21.15.142:6803 osd.0 since back 2024-09-06T14:30:24.605306+0000 front 2024-09-06T14:30:24.605599+0000 (oldest deadline 2024-09-06T14:30:48.705372+0000) 2024-09-06T14:30:59.638 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 06 14:30:59 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-6[49599]: 2024-09-06T14:30:59.412+0000 7fef2ba64640 -1 osd.6 64 heartbeat_check: no reply from 172.21.15.142:6803 osd.0 since back 2024-09-06T14:30:25.822692+0000 front 2024-09-06T14:30:25.822628+0000 (oldest deadline 2024-09-06T14:30:51.122413+0000) 2024-09-06T14:30:59.638 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 06 14:30:59 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-6[49599]: 2024-09-06T14:30:59.412+0000 7fef2ba64640 -1 osd.6 64 heartbeat_check: no reply from 172.21.15.142:6807 osd.1 since back 2024-09-06T14:30:31.122845+0000 front 2024-09-06T14:30:31.122767+0000 (oldest deadline 2024-09-06T14:30:57.022798+0000) 2024-09-06T14:31:00.048 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 06 14:30:59 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-5[44099]: 2024-09-06T14:30:59.640+0000 7fbac27a0640 -1 osd.5 64 heartbeat_check: no reply from 172.21.15.142:6803 osd.0 since back 2024-09-06T14:30:24.533858+0000 front 2024-09-06T14:30:24.534023+0000 (oldest deadline 2024-09-06T14:30:50.433941+0000) 2024-09-06T14:31:00.048 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 06 14:30:59 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-5[44099]: 2024-09-06T14:30:59.640+0000 7fbac27a0640 -1 osd.5 64 heartbeat_check: no reply from 172.21.15.142:6807 osd.1 since back 2024-09-06T14:30:33.334947+0000 front 2024-09-06T14:30:33.335179+0000 (oldest deadline 2024-09-06T14:30:56.835064+0000) 2024-09-06T14:31:00.048 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:30:59 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7[55156]: 2024-09-06T14:30:59.828+0000 7fa8a1380640 -1 osd.7 64 heartbeat_check: no reply from 172.21.15.142:6803 osd.0 since back 2024-09-06T14:30:25.832410+0000 front 2024-09-06T14:30:25.832610+0000 (oldest deadline 2024-09-06T14:30:49.332618+0000) 2024-09-06T14:31:00.048 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:30:59 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7[55156]: 2024-09-06T14:30:59.828+0000 7fa8a1380640 -1 osd.7 64 heartbeat_check: no reply from 172.21.15.142:6807 osd.1 since back 2024-09-06T14:30:31.633604+0000 front 2024-09-06T14:30:31.633572+0000 (oldest deadline 2024-09-06T14:30:56.933451+0000) 2024-09-06T14:31:00.632 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 06 14:31:00 smithi162 podman[66090]: 2024-09-06 14:31:00.328821416 +0000 UTC m=+5.525482119 container died 6a76a4b34e32499cad74ed477d9f796dd406725135f5f0564a794ba511689fb2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c86edd48571ab22373283621eb9feebce9df9c291d4164cfc4df1a09493e1f30, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-4, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-athakkar-testing-2024-09-04-1837-8b38c33, org.label-schema.vendor=CentOS) 2024-09-06T14:31:00.632 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 06 14:31:00 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-6[49599]: 2024-09-06T14:31:00.446+0000 7fef2ba64640 -1 osd.6 64 heartbeat_check: no reply from 172.21.15.142:6803 osd.0 since back 2024-09-06T14:30:25.822692+0000 front 2024-09-06T14:30:25.822628+0000 (oldest deadline 2024-09-06T14:30:51.122413+0000) 2024-09-06T14:31:00.633 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 06 14:31:00 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-6[49599]: 2024-09-06T14:31:00.446+0000 7fef2ba64640 -1 osd.6 64 heartbeat_check: no reply from 172.21.15.142:6807 osd.1 since back 2024-09-06T14:30:31.122845+0000 front 2024-09-06T14:30:31.122767+0000 (oldest deadline 2024-09-06T14:30:57.022798+0000) 2024-09-06T14:31:00.974 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 06 14:31:00 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-5[44099]: 2024-09-06T14:31:00.684+0000 7fbac27a0640 -1 osd.5 64 heartbeat_check: no reply from 172.21.15.142:6803 osd.0 since back 2024-09-06T14:30:24.533858+0000 front 2024-09-06T14:30:24.534023+0000 (oldest deadline 2024-09-06T14:30:50.433941+0000) 2024-09-06T14:31:00.974 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 06 14:31:00 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-5[44099]: 2024-09-06T14:31:00.684+0000 7fbac27a0640 -1 osd.5 64 heartbeat_check: no reply from 172.21.15.142:6807 osd.1 since back 2024-09-06T14:30:33.334947+0000 front 2024-09-06T14:30:33.335179+0000 (oldest deadline 2024-09-06T14:30:56.835064+0000) 2024-09-06T14:31:00.975 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 06 14:31:00 smithi162 podman[66090]: 2024-09-06 14:31:00.792324086 +0000 UTC m=+5.988984829 container cleanup 6a76a4b34e32499cad74ed477d9f796dd406725135f5f0564a794ba511689fb2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c86edd48571ab22373283621eb9feebce9df9c291d4164cfc4df1a09493e1f30, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-4, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, RELEASE=wip-athakkar-testing-2024-09-04-1837-8b38c33, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-06T14:31:00.975 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 06 14:31:00 smithi162 bash[66090]: ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-4 2024-09-06T14:31:00.975 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:31:00 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7[55156]: 2024-09-06T14:31:00.778+0000 7fa8a1380640 -1 osd.7 64 heartbeat_check: no reply from 172.21.15.142:6803 osd.0 since back 2024-09-06T14:30:25.832410+0000 front 2024-09-06T14:30:25.832610+0000 (oldest deadline 2024-09-06T14:30:49.332618+0000) 2024-09-06T14:31:00.975 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:31:00 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7[55156]: 2024-09-06T14:31:00.778+0000 7fa8a1380640 -1 osd.7 64 heartbeat_check: no reply from 172.21.15.142:6807 osd.1 since back 2024-09-06T14:30:31.633604+0000 front 2024-09-06T14:30:31.633572+0000 (oldest deadline 2024-09-06T14:30:56.933451+0000) 2024-09-06T14:31:01.297 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 06 14:31:00 smithi162 podman[66105]: 2024-09-06 14:31:00.977229656 +0000 UTC m=+0.645126419 container remove 6a76a4b34e32499cad74ed477d9f796dd406725135f5f0564a794ba511689fb2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c86edd48571ab22373283621eb9feebce9df9c291d4164cfc4df1a09493e1f30, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-4, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-athakkar-testing-2024-09-04-1837-8b38c33, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.build-date=20240731) 2024-09-06T14:31:01.653 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 06 14:31:01 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-6[49599]: 2024-09-06T14:31:01.403+0000 7fef2ba64640 -1 osd.6 64 heartbeat_check: no reply from 172.21.15.142:6803 osd.0 since back 2024-09-06T14:30:25.822692+0000 front 2024-09-06T14:30:25.822628+0000 (oldest deadline 2024-09-06T14:30:51.122413+0000) 2024-09-06T14:31:01.653 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 06 14:31:01 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-6[49599]: 2024-09-06T14:31:01.403+0000 7fef2ba64640 -1 osd.6 64 heartbeat_check: no reply from 172.21.15.142:6807 osd.1 since back 2024-09-06T14:30:31.122845+0000 front 2024-09-06T14:30:31.122767+0000 (oldest deadline 2024-09-06T14:30:57.022798+0000) 2024-09-06T14:31:01.653 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 06 14:31:01 smithi162 podman[66192]: 2024-09-06 14:31:01.319335048 +0000 UTC m=+0.016009082 image pull 1c45aca0610a312233aa2613f7b416b2884ae14779f1297041485ab28f1c8110 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c86edd48571ab22373283621eb9feebce9df9c291d4164cfc4df1a09493e1f30 2024-09-06T14:31:01.653 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 06 14:31:01 smithi162 podman[66192]: 2024-09-06 14:31:01.428210555 +0000 UTC m=+0.124884587 container create e66ece128c99d6bee3849dc364272fcc7b0d5cfae1169d9f7017b9431ca33d9f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c86edd48571ab22373283621eb9feebce9df9c291d4164cfc4df1a09493e1f30, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-4-deactivate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-athakkar-testing-2024-09-04-1837-8b38c33, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-06T14:31:01.653 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 06 14:31:01 smithi162 podman[66192]: 2024-09-06 14:31:01.656188345 +0000 UTC m=+0.352862378 container init e66ece128c99d6bee3849dc364272fcc7b0d5cfae1169d9f7017b9431ca33d9f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c86edd48571ab22373283621eb9feebce9df9c291d4164cfc4df1a09493e1f30, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-4-deactivate, RELEASE=wip-athakkar-testing-2024-09-04-1837-8b38c33, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True) 2024-09-06T14:31:02.048 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:31:01 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7[55156]: 2024-09-06T14:31:01.749+0000 7fa8a1380640 -1 osd.7 64 heartbeat_check: no reply from 172.21.15.142:6803 osd.0 since back 2024-09-06T14:30:25.832410+0000 front 2024-09-06T14:30:25.832610+0000 (oldest deadline 2024-09-06T14:30:49.332618+0000) 2024-09-06T14:31:02.048 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:31:01 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7[55156]: 2024-09-06T14:31:01.749+0000 7fa8a1380640 -1 osd.7 64 heartbeat_check: no reply from 172.21.15.142:6807 osd.1 since back 2024-09-06T14:30:31.633604+0000 front 2024-09-06T14:30:31.633572+0000 (oldest deadline 2024-09-06T14:30:56.933451+0000) 2024-09-06T14:31:02.048 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 06 14:31:01 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-5[44099]: 2024-09-06T14:31:01.696+0000 7fbac27a0640 -1 osd.5 64 heartbeat_check: no reply from 172.21.15.142:6803 osd.0 since back 2024-09-06T14:30:24.533858+0000 front 2024-09-06T14:30:24.534023+0000 (oldest deadline 2024-09-06T14:30:50.433941+0000) 2024-09-06T14:31:02.048 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 06 14:31:01 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-5[44099]: 2024-09-06T14:31:01.696+0000 7fbac27a0640 -1 osd.5 64 heartbeat_check: no reply from 172.21.15.142:6807 osd.1 since back 2024-09-06T14:30:33.334947+0000 front 2024-09-06T14:30:33.335179+0000 (oldest deadline 2024-09-06T14:30:56.835064+0000) 2024-09-06T14:31:02.049 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 06 14:31:01 smithi162 podman[66192]: 2024-09-06 14:31:01.659912258 +0000 UTC m=+0.356586299 container start e66ece128c99d6bee3849dc364272fcc7b0d5cfae1169d9f7017b9431ca33d9f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c86edd48571ab22373283621eb9feebce9df9c291d4164cfc4df1a09493e1f30, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-4-deactivate, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-athakkar-testing-2024-09-04-1837-8b38c33, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-09-06T14:31:02.049 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 06 14:31:01 smithi162 podman[66192]: 2024-09-06 14:31:01.77311591 +0000 UTC m=+0.469789947 container attach e66ece128c99d6bee3849dc364272fcc7b0d5cfae1169d9f7017b9431ca33d9f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c86edd48571ab22373283621eb9feebce9df9c291d4164cfc4df1a09493e1f30, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-4-deactivate, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, RELEASE=wip-athakkar-testing-2024-09-04-1837-8b38c33, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-06T14:31:02.049 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 06 14:31:01 smithi162 podman[66192]: 2024-09-06 14:31:01.849479344 +0000 UTC m=+0.546153378 container died e66ece128c99d6bee3849dc364272fcc7b0d5cfae1169d9f7017b9431ca33d9f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c86edd48571ab22373283621eb9feebce9df9c291d4164cfc4df1a09493e1f30, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, RELEASE=wip-athakkar-testing-2024-09-04-1837-8b38c33, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.37.2) 2024-09-06T14:31:02.499 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 06 14:31:02 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-6[49599]: 2024-09-06T14:31:02.387+0000 7fef2ba64640 -1 osd.6 64 heartbeat_check: no reply from 172.21.15.142:6803 osd.0 since back 2024-09-06T14:30:25.822692+0000 front 2024-09-06T14:30:25.822628+0000 (oldest deadline 2024-09-06T14:30:51.122413+0000) 2024-09-06T14:31:02.499 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 06 14:31:02 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-6[49599]: 2024-09-06T14:31:02.387+0000 7fef2ba64640 -1 osd.6 64 heartbeat_check: no reply from 172.21.15.142:6807 osd.1 since back 2024-09-06T14:30:31.122845+0000 front 2024-09-06T14:30:31.122767+0000 (oldest deadline 2024-09-06T14:30:57.022798+0000) 2024-09-06T14:31:02.529 DEBUG:teuthology.orchestra.run.smithi162:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ce32462e-6c5a-11ef-bce4-c7b262605968@osd.4.service' 2024-09-06T14:31:02.571 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-06T14:31:02.571 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-09-06T14:31:02.571 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-09-06T14:31:02.571 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ceph-ce32462e-6c5a-11ef-bce4-c7b262605968@osd.5 2024-09-06T14:31:02.793 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 06 14:31:02 smithi162 systemd[1]: Stopping Ceph osd.5 for ce32462e-6c5a-11ef-bce4-c7b262605968... 2024-09-06T14:31:02.794 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 06 14:31:02 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-5[44099]: 2024-09-06T14:31:02.717+0000 7fbac27a0640 -1 osd.5 64 heartbeat_check: no reply from 172.21.15.142:6803 osd.0 since back 2024-09-06T14:30:24.533858+0000 front 2024-09-06T14:30:24.534023+0000 (oldest deadline 2024-09-06T14:30:50.433941+0000) 2024-09-06T14:31:02.794 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 06 14:31:02 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-5[44099]: 2024-09-06T14:31:02.717+0000 7fbac27a0640 -1 osd.5 64 heartbeat_check: no reply from 172.21.15.142:6807 osd.1 since back 2024-09-06T14:30:33.334947+0000 front 2024-09-06T14:30:33.335179+0000 (oldest deadline 2024-09-06T14:30:56.835064+0000) 2024-09-06T14:31:02.794 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:31:02 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7[55156]: 2024-09-06T14:31:02.739+0000 7fa8a1380640 -1 osd.7 64 heartbeat_check: no reply from 172.21.15.142:6803 osd.0 since back 2024-09-06T14:30:25.832410+0000 front 2024-09-06T14:30:25.832610+0000 (oldest deadline 2024-09-06T14:30:49.332618+0000) 2024-09-06T14:31:02.794 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:31:02 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7[55156]: 2024-09-06T14:31:02.739+0000 7fa8a1380640 -1 osd.7 64 heartbeat_check: no reply from 172.21.15.142:6807 osd.1 since back 2024-09-06T14:30:31.633604+0000 front 2024-09-06T14:30:31.633572+0000 (oldest deadline 2024-09-06T14:30:56.933451+0000) 2024-09-06T14:31:03.048 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 06 14:31:02 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-5[44099]: 2024-09-06T14:31:02.929+0000 7fbaca193640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-06T14:31:03.048 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 06 14:31:02 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-5[44099]: 2024-09-06T14:31:02.929+0000 7fbaca193640 -1 osd.5 64 *** Got signal Terminated *** 2024-09-06T14:31:03.048 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 06 14:31:02 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-5[44099]: 2024-09-06T14:31:02.929+0000 7fbaca193640 -1 osd.5 64 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-06T14:31:03.548 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 06 14:31:03 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-6[49599]: 2024-09-06T14:31:03.346+0000 7fef2ba64640 -1 osd.6 64 heartbeat_check: no reply from 172.21.15.142:6803 osd.0 since back 2024-09-06T14:30:25.822692+0000 front 2024-09-06T14:30:25.822628+0000 (oldest deadline 2024-09-06T14:30:51.122413+0000) 2024-09-06T14:31:03.548 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 06 14:31:03 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-6[49599]: 2024-09-06T14:31:03.346+0000 7fef2ba64640 -1 osd.6 64 heartbeat_check: no reply from 172.21.15.142:6807 osd.1 since back 2024-09-06T14:30:31.122845+0000 front 2024-09-06T14:30:31.122767+0000 (oldest deadline 2024-09-06T14:30:57.022798+0000) 2024-09-06T14:31:03.548 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 06 14:31:03 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-prometheus-a[64015]: ts=2024-09-06T14:31:03.180Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=nvmeof msg="Unable to refresh target groups" err="Get \"http://172.21.15.142:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.142:8765: connect: connection refused" 2024-09-06T14:31:03.548 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 06 14:31:03 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-prometheus-a[64015]: ts=2024-09-06T14:31:03.181Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.142:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.142:8765: connect: connection refused" 2024-09-06T14:31:03.548 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 06 14:31:03 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-prometheus-a[64015]: ts=2024-09-06T14:31:03.183Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.142:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.142:8765: connect: connection refused" 2024-09-06T14:31:03.548 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 06 14:31:03 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-prometheus-a[64015]: ts=2024-09-06T14:31:03.183Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=nfs msg="Unable to refresh target groups" err="Get \"http://172.21.15.142:8765/sd/prometheus/sd-config?service=nfs\": dial tcp 172.21.15.142:8765: connect: connection refused" 2024-09-06T14:31:03.548 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 06 14:31:03 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-prometheus-a[64015]: ts=2024-09-06T14:31:03.184Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.142:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.142:8765: connect: connection refused" 2024-09-06T14:31:03.549 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 06 14:31:03 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-prometheus-a[64015]: ts=2024-09-06T14:31:03.184Z caller=refresh.go:90 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.142:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.142:8765: connect: connection refused" 2024-09-06T14:31:04.048 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 06 14:31:03 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-5[44099]: 2024-09-06T14:31:03.732+0000 7fbac27a0640 -1 osd.5 64 heartbeat_check: no reply from 172.21.15.142:6803 osd.0 since back 2024-09-06T14:30:24.533858+0000 front 2024-09-06T14:30:24.534023+0000 (oldest deadline 2024-09-06T14:30:50.433941+0000) 2024-09-06T14:31:04.048 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 06 14:31:03 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-5[44099]: 2024-09-06T14:31:03.732+0000 7fbac27a0640 -1 osd.5 64 heartbeat_check: no reply from 172.21.15.142:6807 osd.1 since back 2024-09-06T14:30:33.334947+0000 front 2024-09-06T14:30:33.335179+0000 (oldest deadline 2024-09-06T14:30:56.835064+0000) 2024-09-06T14:31:04.048 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:31:03 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7[55156]: 2024-09-06T14:31:03.764+0000 7fa8a1380640 -1 osd.7 64 heartbeat_check: no reply from 172.21.15.142:6803 osd.0 since back 2024-09-06T14:30:25.832410+0000 front 2024-09-06T14:30:25.832610+0000 (oldest deadline 2024-09-06T14:30:49.332618+0000) 2024-09-06T14:31:04.048 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:31:03 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7[55156]: 2024-09-06T14:31:03.764+0000 7fa8a1380640 -1 osd.7 64 heartbeat_check: no reply from 172.21.15.142:6807 osd.1 since back 2024-09-06T14:30:31.633604+0000 front 2024-09-06T14:30:31.633572+0000 (oldest deadline 2024-09-06T14:30:56.933451+0000) 2024-09-06T14:31:04.710 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 06 14:31:04 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-6[49599]: 2024-09-06T14:31:04.335+0000 7fef2ba64640 -1 osd.6 64 heartbeat_check: no reply from 172.21.15.142:6803 osd.0 since back 2024-09-06T14:30:25.822692+0000 front 2024-09-06T14:30:25.822628+0000 (oldest deadline 2024-09-06T14:30:51.122413+0000) 2024-09-06T14:31:04.710 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 06 14:31:04 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-6[49599]: 2024-09-06T14:31:04.335+0000 7fef2ba64640 -1 osd.6 64 heartbeat_check: no reply from 172.21.15.142:6807 osd.1 since back 2024-09-06T14:30:31.122845+0000 front 2024-09-06T14:30:31.122767+0000 (oldest deadline 2024-09-06T14:30:57.022798+0000) 2024-09-06T14:31:05.048 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 06 14:31:04 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-5[44099]: 2024-09-06T14:31:04.712+0000 7fbac27a0640 -1 osd.5 64 heartbeat_check: no reply from 172.21.15.142:6803 osd.0 since back 2024-09-06T14:30:24.533858+0000 front 2024-09-06T14:30:24.534023+0000 (oldest deadline 2024-09-06T14:30:50.433941+0000) 2024-09-06T14:31:05.048 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 06 14:31:04 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-5[44099]: 2024-09-06T14:31:04.712+0000 7fbac27a0640 -1 osd.5 64 heartbeat_check: no reply from 172.21.15.142:6807 osd.1 since back 2024-09-06T14:30:33.334947+0000 front 2024-09-06T14:30:33.335179+0000 (oldest deadline 2024-09-06T14:30:56.835064+0000) 2024-09-06T14:31:05.048 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:31:04 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7[55156]: 2024-09-06T14:31:04.724+0000 7fa8a1380640 -1 osd.7 64 heartbeat_check: no reply from 172.21.15.142:6803 osd.0 since back 2024-09-06T14:30:25.832410+0000 front 2024-09-06T14:30:25.832610+0000 (oldest deadline 2024-09-06T14:30:49.332618+0000) 2024-09-06T14:31:05.048 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:31:04 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7[55156]: 2024-09-06T14:31:04.724+0000 7fa8a1380640 -1 osd.7 64 heartbeat_check: no reply from 172.21.15.142:6807 osd.1 since back 2024-09-06T14:30:31.633604+0000 front 2024-09-06T14:30:31.633572+0000 (oldest deadline 2024-09-06T14:30:56.933451+0000) 2024-09-06T14:31:05.685 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 06 14:31:05 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-6[49599]: 2024-09-06T14:31:05.352+0000 7fef2ba64640 -1 osd.6 64 heartbeat_check: no reply from 172.21.15.142:6803 osd.0 since back 2024-09-06T14:30:25.822692+0000 front 2024-09-06T14:30:25.822628+0000 (oldest deadline 2024-09-06T14:30:51.122413+0000) 2024-09-06T14:31:05.685 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 06 14:31:05 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-6[49599]: 2024-09-06T14:31:05.352+0000 7fef2ba64640 -1 osd.6 64 heartbeat_check: no reply from 172.21.15.142:6807 osd.1 since back 2024-09-06T14:30:31.122845+0000 front 2024-09-06T14:30:31.122767+0000 (oldest deadline 2024-09-06T14:30:57.022798+0000) 2024-09-06T14:31:06.048 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 06 14:31:05 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-5[44099]: 2024-09-06T14:31:05.736+0000 7fbac27a0640 -1 osd.5 64 heartbeat_check: no reply from 172.21.15.142:6803 osd.0 since back 2024-09-06T14:30:24.533858+0000 front 2024-09-06T14:30:24.534023+0000 (oldest deadline 2024-09-06T14:30:50.433941+0000) 2024-09-06T14:31:06.048 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 06 14:31:05 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-5[44099]: 2024-09-06T14:31:05.736+0000 7fbac27a0640 -1 osd.5 64 heartbeat_check: no reply from 172.21.15.142:6807 osd.1 since back 2024-09-06T14:30:33.334947+0000 front 2024-09-06T14:30:33.335179+0000 (oldest deadline 2024-09-06T14:30:56.835064+0000) 2024-09-06T14:31:06.048 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 06 14:31:05 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-5[44099]: 2024-09-06T14:31:05.736+0000 7fbac27a0640 -1 osd.5 64 heartbeat_check: no reply from 172.21.15.142:6811 osd.2 since back 2024-09-06T14:30:39.036350+0000 front 2024-09-06T14:30:39.036438+0000 (oldest deadline 2024-09-06T14:31:04.936229+0000) 2024-09-06T14:31:06.048 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:31:05 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7[55156]: 2024-09-06T14:31:05.687+0000 7fa8a1380640 -1 osd.7 64 heartbeat_check: no reply from 172.21.15.142:6803 osd.0 since back 2024-09-06T14:30:25.832410+0000 front 2024-09-06T14:30:25.832610+0000 (oldest deadline 2024-09-06T14:30:49.332618+0000) 2024-09-06T14:31:06.048 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:31:05 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7[55156]: 2024-09-06T14:31:05.687+0000 7fa8a1380640 -1 osd.7 64 heartbeat_check: no reply from 172.21.15.142:6807 osd.1 since back 2024-09-06T14:30:31.633604+0000 front 2024-09-06T14:30:31.633572+0000 (oldest deadline 2024-09-06T14:30:56.933451+0000) 2024-09-06T14:31:06.662 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 06 14:31:06 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-6[49599]: 2024-09-06T14:31:06.332+0000 7fef2ba64640 -1 osd.6 64 heartbeat_check: no reply from 172.21.15.142:6803 osd.0 since back 2024-09-06T14:30:25.822692+0000 front 2024-09-06T14:30:25.822628+0000 (oldest deadline 2024-09-06T14:30:51.122413+0000) 2024-09-06T14:31:06.662 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 06 14:31:06 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-6[49599]: 2024-09-06T14:31:06.332+0000 7fef2ba64640 -1 osd.6 64 heartbeat_check: no reply from 172.21.15.142:6807 osd.1 since back 2024-09-06T14:30:31.122845+0000 front 2024-09-06T14:30:31.122767+0000 (oldest deadline 2024-09-06T14:30:57.022798+0000) 2024-09-06T14:31:07.048 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:31:06 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7[55156]: 2024-09-06T14:31:06.664+0000 7fa8a1380640 -1 osd.7 64 heartbeat_check: no reply from 172.21.15.142:6803 osd.0 since back 2024-09-06T14:30:25.832410+0000 front 2024-09-06T14:30:25.832610+0000 (oldest deadline 2024-09-06T14:30:49.332618+0000) 2024-09-06T14:31:07.048 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:31:06 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7[55156]: 2024-09-06T14:31:06.664+0000 7fa8a1380640 -1 osd.7 64 heartbeat_check: no reply from 172.21.15.142:6807 osd.1 since back 2024-09-06T14:30:31.633604+0000 front 2024-09-06T14:30:31.633572+0000 (oldest deadline 2024-09-06T14:30:56.933451+0000) 2024-09-06T14:31:07.048 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 06 14:31:06 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-5[44099]: 2024-09-06T14:31:06.766+0000 7fbac27a0640 -1 osd.5 64 heartbeat_check: no reply from 172.21.15.142:6803 osd.0 since back 2024-09-06T14:30:24.533858+0000 front 2024-09-06T14:30:24.534023+0000 (oldest deadline 2024-09-06T14:30:50.433941+0000) 2024-09-06T14:31:07.048 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 06 14:31:06 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-5[44099]: 2024-09-06T14:31:06.766+0000 7fbac27a0640 -1 osd.5 64 heartbeat_check: no reply from 172.21.15.142:6807 osd.1 since back 2024-09-06T14:30:33.334947+0000 front 2024-09-06T14:30:33.335179+0000 (oldest deadline 2024-09-06T14:30:56.835064+0000) 2024-09-06T14:31:07.048 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 06 14:31:06 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-5[44099]: 2024-09-06T14:31:06.766+0000 7fbac27a0640 -1 osd.5 64 heartbeat_check: no reply from 172.21.15.142:6811 osd.2 since back 2024-09-06T14:30:39.036350+0000 front 2024-09-06T14:30:39.036438+0000 (oldest deadline 2024-09-06T14:31:04.936229+0000) 2024-09-06T14:31:07.548 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 06 14:31:07 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-6[49599]: 2024-09-06T14:31:07.288+0000 7fef2ba64640 -1 osd.6 64 heartbeat_check: no reply from 172.21.15.142:6803 osd.0 since back 2024-09-06T14:30:25.822692+0000 front 2024-09-06T14:30:25.822628+0000 (oldest deadline 2024-09-06T14:30:51.122413+0000) 2024-09-06T14:31:07.548 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 06 14:31:07 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-6[49599]: 2024-09-06T14:31:07.289+0000 7fef2ba64640 -1 osd.6 64 heartbeat_check: no reply from 172.21.15.142:6807 osd.1 since back 2024-09-06T14:30:31.122845+0000 front 2024-09-06T14:30:31.122767+0000 (oldest deadline 2024-09-06T14:30:57.022798+0000) 2024-09-06T14:31:07.548 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 06 14:31:07 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-6[49599]: 2024-09-06T14:31:07.289+0000 7fef2ba64640 -1 osd.6 64 heartbeat_check: no reply from 172.21.15.142:6811 osd.2 since back 2024-09-06T14:30:42.923955+0000 front 2024-09-06T14:30:42.923941+0000 (oldest deadline 2024-09-06T14:31:06.423830+0000) 2024-09-06T14:31:08.048 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 06 14:31:07 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-5[44099]: 2024-09-06T14:31:07.786+0000 7fbac27a0640 -1 osd.5 64 heartbeat_check: no reply from 172.21.15.142:6803 osd.0 since back 2024-09-06T14:30:24.533858+0000 front 2024-09-06T14:30:24.534023+0000 (oldest deadline 2024-09-06T14:30:50.433941+0000) 2024-09-06T14:31:08.048 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 06 14:31:07 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-5[44099]: 2024-09-06T14:31:07.786+0000 7fbac27a0640 -1 osd.5 64 heartbeat_check: no reply from 172.21.15.142:6807 osd.1 since back 2024-09-06T14:30:33.334947+0000 front 2024-09-06T14:30:33.335179+0000 (oldest deadline 2024-09-06T14:30:56.835064+0000) 2024-09-06T14:31:08.049 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 06 14:31:07 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-5[44099]: 2024-09-06T14:31:07.786+0000 7fbac27a0640 -1 osd.5 64 heartbeat_check: no reply from 172.21.15.142:6811 osd.2 since back 2024-09-06T14:30:39.036350+0000 front 2024-09-06T14:30:39.036438+0000 (oldest deadline 2024-09-06T14:31:04.936229+0000) 2024-09-06T14:31:08.049 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:31:07 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7[55156]: 2024-09-06T14:31:07.713+0000 7fa8a1380640 -1 osd.7 64 heartbeat_check: no reply from 172.21.15.142:6803 osd.0 since back 2024-09-06T14:30:25.832410+0000 front 2024-09-06T14:30:25.832610+0000 (oldest deadline 2024-09-06T14:30:49.332618+0000) 2024-09-06T14:31:08.049 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:31:07 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7[55156]: 2024-09-06T14:31:07.713+0000 7fa8a1380640 -1 osd.7 64 heartbeat_check: no reply from 172.21.15.142:6807 osd.1 since back 2024-09-06T14:30:31.633604+0000 front 2024-09-06T14:30:31.633572+0000 (oldest deadline 2024-09-06T14:30:56.933451+0000) 2024-09-06T14:31:08.049 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:31:07 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7[55156]: 2024-09-06T14:31:07.713+0000 7fa8a1380640 -1 osd.7 64 heartbeat_check: no reply from 172.21.15.142:6811 osd.2 since back 2024-09-06T14:30:42.134879+0000 front 2024-09-06T14:30:42.134961+0000 (oldest deadline 2024-09-06T14:31:06.834776+0000) 2024-09-06T14:31:08.548 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 06 14:31:08 smithi162 podman[66295]: 2024-09-06 14:31:08.388109305 +0000 UTC m=+5.579905370 container stop ebda6095c9ae6960e7043af663bf200b89fea343a33c2fc98e0c972d324c87ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c86edd48571ab22373283621eb9feebce9df9c291d4164cfc4df1a09493e1f30, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-5, org.label-schema.schema-version=1.0, RELEASE=wip-athakkar-testing-2024-09-04-1837-8b38c33, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, ceph=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True) 2024-09-06T14:31:08.548 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 06 14:31:08 smithi162 podman[66295]: 2024-09-06 14:31:08.388690263 +0000 UTC m=+5.580486328 container died ebda6095c9ae6960e7043af663bf200b89fea343a33c2fc98e0c972d324c87ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c86edd48571ab22373283621eb9feebce9df9c291d4164cfc4df1a09493e1f30, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, ceph=True, RELEASE=wip-athakkar-testing-2024-09-04-1837-8b38c33, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731) 2024-09-06T14:31:08.549 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 06 14:31:08 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-6[49599]: 2024-09-06T14:31:08.249+0000 7fef2ba64640 -1 osd.6 64 heartbeat_check: no reply from 172.21.15.142:6803 osd.0 since back 2024-09-06T14:30:25.822692+0000 front 2024-09-06T14:30:25.822628+0000 (oldest deadline 2024-09-06T14:30:51.122413+0000) 2024-09-06T14:31:08.549 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 06 14:31:08 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-6[49599]: 2024-09-06T14:31:08.249+0000 7fef2ba64640 -1 osd.6 64 heartbeat_check: no reply from 172.21.15.142:6807 osd.1 since back 2024-09-06T14:30:31.122845+0000 front 2024-09-06T14:30:31.122767+0000 (oldest deadline 2024-09-06T14:30:57.022798+0000) 2024-09-06T14:31:08.549 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 06 14:31:08 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-6[49599]: 2024-09-06T14:31:08.249+0000 7fef2ba64640 -1 osd.6 64 heartbeat_check: no reply from 172.21.15.142:6811 osd.2 since back 2024-09-06T14:30:42.923955+0000 front 2024-09-06T14:30:42.923941+0000 (oldest deadline 2024-09-06T14:31:06.423830+0000) 2024-09-06T14:31:09.045 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 06 14:31:08 smithi162 podman[66295]: 2024-09-06 14:31:08.771716003 +0000 UTC m=+5.963512065 container cleanup ebda6095c9ae6960e7043af663bf200b89fea343a33c2fc98e0c972d324c87ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c86edd48571ab22373283621eb9feebce9df9c291d4164cfc4df1a09493e1f30, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-athakkar-testing-2024-09-04-1837-8b38c33, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2) 2024-09-06T14:31:09.045 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 06 14:31:08 smithi162 bash[66295]: ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-5 2024-09-06T14:31:09.045 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:31:08 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7[55156]: 2024-09-06T14:31:08.679+0000 7fa8a1380640 -1 osd.7 64 heartbeat_check: no reply from 172.21.15.142:6803 osd.0 since back 2024-09-06T14:30:25.832410+0000 front 2024-09-06T14:30:25.832610+0000 (oldest deadline 2024-09-06T14:30:49.332618+0000) 2024-09-06T14:31:09.045 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:31:08 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7[55156]: 2024-09-06T14:31:08.679+0000 7fa8a1380640 -1 osd.7 64 heartbeat_check: no reply from 172.21.15.142:6807 osd.1 since back 2024-09-06T14:30:31.633604+0000 front 2024-09-06T14:30:31.633572+0000 (oldest deadline 2024-09-06T14:30:56.933451+0000) 2024-09-06T14:31:09.045 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:31:08 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7[55156]: 2024-09-06T14:31:08.679+0000 7fa8a1380640 -1 osd.7 64 heartbeat_check: no reply from 172.21.15.142:6811 osd.2 since back 2024-09-06T14:30:42.134879+0000 front 2024-09-06T14:30:42.134961+0000 (oldest deadline 2024-09-06T14:31:06.834776+0000) 2024-09-06T14:31:09.298 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 06 14:31:09 smithi162 podman[66321]: 2024-09-06 14:31:09.047941061 +0000 UTC m=+0.651064468 container remove ebda6095c9ae6960e7043af663bf200b89fea343a33c2fc98e0c972d324c87ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c86edd48571ab22373283621eb9feebce9df9c291d4164cfc4df1a09493e1f30, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-5, org.label-schema.build-date=20240731, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-athakkar-testing-2024-09-04-1837-8b38c33, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-06T14:31:09.298 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 06 14:31:09 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-6[49599]: 2024-09-06T14:31:09.233+0000 7fef2ba64640 -1 osd.6 64 heartbeat_check: no reply from 172.21.15.142:6803 osd.0 since back 2024-09-06T14:30:25.822692+0000 front 2024-09-06T14:30:25.822628+0000 (oldest deadline 2024-09-06T14:30:51.122413+0000) 2024-09-06T14:31:09.298 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 06 14:31:09 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-6[49599]: 2024-09-06T14:31:09.233+0000 7fef2ba64640 -1 osd.6 64 heartbeat_check: no reply from 172.21.15.142:6807 osd.1 since back 2024-09-06T14:30:31.122845+0000 front 2024-09-06T14:30:31.122767+0000 (oldest deadline 2024-09-06T14:30:57.022798+0000) 2024-09-06T14:31:09.298 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 06 14:31:09 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-6[49599]: 2024-09-06T14:31:09.233+0000 7fef2ba64640 -1 osd.6 64 heartbeat_check: no reply from 172.21.15.142:6811 osd.2 since back 2024-09-06T14:30:42.923955+0000 front 2024-09-06T14:30:42.923941+0000 (oldest deadline 2024-09-06T14:31:06.423830+0000) 2024-09-06T14:31:09.673 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:31:09 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7[55156]: 2024-09-06T14:31:09.631+0000 7fa8a1380640 -1 osd.7 64 heartbeat_check: no reply from 172.21.15.142:6803 osd.0 since back 2024-09-06T14:30:25.832410+0000 front 2024-09-06T14:30:25.832610+0000 (oldest deadline 2024-09-06T14:30:49.332618+0000) 2024-09-06T14:31:09.673 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:31:09 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7[55156]: 2024-09-06T14:31:09.631+0000 7fa8a1380640 -1 osd.7 64 heartbeat_check: no reply from 172.21.15.142:6807 osd.1 since back 2024-09-06T14:30:31.633604+0000 front 2024-09-06T14:30:31.633572+0000 (oldest deadline 2024-09-06T14:30:56.933451+0000) 2024-09-06T14:31:09.673 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:31:09 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7[55156]: 2024-09-06T14:31:09.631+0000 7fa8a1380640 -1 osd.7 64 heartbeat_check: no reply from 172.21.15.142:6811 osd.2 since back 2024-09-06T14:30:42.134879+0000 front 2024-09-06T14:30:42.134961+0000 (oldest deadline 2024-09-06T14:31:06.834776+0000) 2024-09-06T14:31:09.674 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 06 14:31:09 smithi162 podman[66414]: 2024-09-06 14:31:09.293021002 +0000 UTC m=+0.015880029 image pull 1c45aca0610a312233aa2613f7b416b2884ae14779f1297041485ab28f1c8110 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c86edd48571ab22373283621eb9feebce9df9c291d4164cfc4df1a09493e1f30 2024-09-06T14:31:09.674 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 06 14:31:09 smithi162 podman[66414]: 2024-09-06 14:31:09.390528075 +0000 UTC m=+0.113387093 container create 73c4cbef47153c2ccc8858abae986b9926f99657889b095c100906406a36c158 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c86edd48571ab22373283621eb9feebce9df9c291d4164cfc4df1a09493e1f30, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-5-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-athakkar-testing-2024-09-04-1837-8b38c33, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731) 2024-09-06T14:31:09.674 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 06 14:31:09 smithi162 podman[66414]: 2024-09-06 14:31:09.601413179 +0000 UTC m=+0.324272196 container init 73c4cbef47153c2ccc8858abae986b9926f99657889b095c100906406a36c158 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c86edd48571ab22373283621eb9feebce9df9c291d4164cfc4df1a09493e1f30, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-5-deactivate, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, RELEASE=wip-athakkar-testing-2024-09-04-1837-8b38c33) 2024-09-06T14:31:09.674 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 06 14:31:09 smithi162 podman[66414]: 2024-09-06 14:31:09.604922847 +0000 UTC m=+0.327781872 container start 73c4cbef47153c2ccc8858abae986b9926f99657889b095c100906406a36c158 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c86edd48571ab22373283621eb9feebce9df9c291d4164cfc4df1a09493e1f30, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-5-deactivate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-athakkar-testing-2024-09-04-1837-8b38c33, CEPH_POINT_RELEASE=) 2024-09-06T14:31:09.674 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 06 14:31:09 smithi162 podman[66414]: 2024-09-06 14:31:09.676407475 +0000 UTC m=+0.399266517 container attach 73c4cbef47153c2ccc8858abae986b9926f99657889b095c100906406a36c158 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c86edd48571ab22373283621eb9feebce9df9c291d4164cfc4df1a09493e1f30, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-5-deactivate, org.label-schema.build-date=20240731, RELEASE=wip-athakkar-testing-2024-09-04-1837-8b38c33, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-06T14:31:10.047 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 06 14:31:09 smithi162 podman[66414]: 2024-09-06 14:31:09.79423661 +0000 UTC m=+0.517095628 container died 73c4cbef47153c2ccc8858abae986b9926f99657889b095c100906406a36c158 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c86edd48571ab22373283621eb9feebce9df9c291d4164cfc4df1a09493e1f30, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-5-deactivate, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-athakkar-testing-2024-09-04-1837-8b38c33, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD) 2024-09-06T14:31:10.385 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 06 14:31:10 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-6[49599]: 2024-09-06T14:31:10.201+0000 7fef2ba64640 -1 osd.6 64 heartbeat_check: no reply from 172.21.15.142:6803 osd.0 since back 2024-09-06T14:30:25.822692+0000 front 2024-09-06T14:30:25.822628+0000 (oldest deadline 2024-09-06T14:30:51.122413+0000) 2024-09-06T14:31:10.385 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 06 14:31:10 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-6[49599]: 2024-09-06T14:31:10.201+0000 7fef2ba64640 -1 osd.6 64 heartbeat_check: no reply from 172.21.15.142:6807 osd.1 since back 2024-09-06T14:30:31.122845+0000 front 2024-09-06T14:30:31.122767+0000 (oldest deadline 2024-09-06T14:30:57.022798+0000) 2024-09-06T14:31:10.386 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 06 14:31:10 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-6[49599]: 2024-09-06T14:31:10.201+0000 7fef2ba64640 -1 osd.6 64 heartbeat_check: no reply from 172.21.15.142:6811 osd.2 since back 2024-09-06T14:30:42.923955+0000 front 2024-09-06T14:30:42.923941+0000 (oldest deadline 2024-09-06T14:31:06.423830+0000) 2024-09-06T14:31:10.407 DEBUG:teuthology.orchestra.run.smithi162:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ce32462e-6c5a-11ef-bce4-c7b262605968@osd.5.service' 2024-09-06T14:31:10.446 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-06T14:31:10.446 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-09-06T14:31:10.447 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-09-06T14:31:10.447 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ceph-ce32462e-6c5a-11ef-bce4-c7b262605968@osd.6 2024-09-06T14:31:10.676 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 06 14:31:10 smithi162 systemd[1]: Stopping Ceph osd.6 for ce32462e-6c5a-11ef-bce4-c7b262605968... 2024-09-06T14:31:11.048 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 06 14:31:10 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-6[49599]: 2024-09-06T14:31:10.775+0000 7fef2f44f640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-06T14:31:11.048 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 06 14:31:10 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-6[49599]: 2024-09-06T14:31:10.775+0000 7fef2f44f640 -1 osd.6 64 *** Got signal Terminated *** 2024-09-06T14:31:11.048 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 06 14:31:10 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-6[49599]: 2024-09-06T14:31:10.775+0000 7fef2f44f640 -1 osd.6 64 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-06T14:31:11.048 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:31:10 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7[55156]: 2024-09-06T14:31:10.678+0000 7fa8a1380640 -1 osd.7 64 heartbeat_check: no reply from 172.21.15.142:6803 osd.0 since back 2024-09-06T14:30:25.832410+0000 front 2024-09-06T14:30:25.832610+0000 (oldest deadline 2024-09-06T14:30:49.332618+0000) 2024-09-06T14:31:11.048 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:31:10 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7[55156]: 2024-09-06T14:31:10.678+0000 7fa8a1380640 -1 osd.7 64 heartbeat_check: no reply from 172.21.15.142:6807 osd.1 since back 2024-09-06T14:30:31.633604+0000 front 2024-09-06T14:30:31.633572+0000 (oldest deadline 2024-09-06T14:30:56.933451+0000) 2024-09-06T14:31:11.048 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:31:10 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7[55156]: 2024-09-06T14:31:10.678+0000 7fa8a1380640 -1 osd.7 64 heartbeat_check: no reply from 172.21.15.142:6811 osd.2 since back 2024-09-06T14:30:42.134879+0000 front 2024-09-06T14:30:42.134961+0000 (oldest deadline 2024-09-06T14:31:06.834776+0000) 2024-09-06T14:31:11.548 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 06 14:31:11 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-6[49599]: 2024-09-06T14:31:11.211+0000 7fef2ba64640 -1 osd.6 64 heartbeat_check: no reply from 172.21.15.142:6803 osd.0 since back 2024-09-06T14:30:25.822692+0000 front 2024-09-06T14:30:25.822628+0000 (oldest deadline 2024-09-06T14:30:51.122413+0000) 2024-09-06T14:31:11.548 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 06 14:31:11 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-6[49599]: 2024-09-06T14:31:11.211+0000 7fef2ba64640 -1 osd.6 64 heartbeat_check: no reply from 172.21.15.142:6807 osd.1 since back 2024-09-06T14:30:31.122845+0000 front 2024-09-06T14:30:31.122767+0000 (oldest deadline 2024-09-06T14:30:57.022798+0000) 2024-09-06T14:31:11.548 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 06 14:31:11 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-6[49599]: 2024-09-06T14:31:11.211+0000 7fef2ba64640 -1 osd.6 64 heartbeat_check: no reply from 172.21.15.142:6811 osd.2 since back 2024-09-06T14:30:42.923955+0000 front 2024-09-06T14:30:42.923941+0000 (oldest deadline 2024-09-06T14:31:06.423830+0000) 2024-09-06T14:31:12.048 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:31:11 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7[55156]: 2024-09-06T14:31:11.725+0000 7fa8a1380640 -1 osd.7 64 heartbeat_check: no reply from 172.21.15.142:6803 osd.0 since back 2024-09-06T14:30:25.832410+0000 front 2024-09-06T14:30:25.832610+0000 (oldest deadline 2024-09-06T14:30:49.332618+0000) 2024-09-06T14:31:12.048 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:31:11 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7[55156]: 2024-09-06T14:31:11.725+0000 7fa8a1380640 -1 osd.7 64 heartbeat_check: no reply from 172.21.15.142:6807 osd.1 since back 2024-09-06T14:30:31.633604+0000 front 2024-09-06T14:30:31.633572+0000 (oldest deadline 2024-09-06T14:30:56.933451+0000) 2024-09-06T14:31:12.048 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:31:11 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7[55156]: 2024-09-06T14:31:11.725+0000 7fa8a1380640 -1 osd.7 64 heartbeat_check: no reply from 172.21.15.142:6811 osd.2 since back 2024-09-06T14:30:42.134879+0000 front 2024-09-06T14:30:42.134961+0000 (oldest deadline 2024-09-06T14:31:06.834776+0000) 2024-09-06T14:31:12.048 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:31:11 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7[55156]: 2024-09-06T14:31:11.725+0000 7fa8a1380640 -1 osd.7 64 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24421.0:364 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64) 2024-09-06T14:31:12.548 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 06 14:31:12 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-6[49599]: 2024-09-06T14:31:12.180+0000 7fef2ba64640 -1 osd.6 64 heartbeat_check: no reply from 172.21.15.142:6803 osd.0 since back 2024-09-06T14:30:25.822692+0000 front 2024-09-06T14:30:25.822628+0000 (oldest deadline 2024-09-06T14:30:51.122413+0000) 2024-09-06T14:31:12.548 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 06 14:31:12 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-6[49599]: 2024-09-06T14:31:12.180+0000 7fef2ba64640 -1 osd.6 64 heartbeat_check: no reply from 172.21.15.142:6807 osd.1 since back 2024-09-06T14:30:31.122845+0000 front 2024-09-06T14:30:31.122767+0000 (oldest deadline 2024-09-06T14:30:57.022798+0000) 2024-09-06T14:31:12.548 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 06 14:31:12 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-6[49599]: 2024-09-06T14:31:12.180+0000 7fef2ba64640 -1 osd.6 64 heartbeat_check: no reply from 172.21.15.142:6811 osd.2 since back 2024-09-06T14:30:42.923955+0000 front 2024-09-06T14:30:42.923941+0000 (oldest deadline 2024-09-06T14:31:06.423830+0000) 2024-09-06T14:31:13.048 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:31:12 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7[55156]: 2024-09-06T14:31:12.721+0000 7fa8a1380640 -1 osd.7 64 heartbeat_check: no reply from 172.21.15.142:6803 osd.0 since back 2024-09-06T14:30:25.832410+0000 front 2024-09-06T14:30:25.832610+0000 (oldest deadline 2024-09-06T14:30:49.332618+0000) 2024-09-06T14:31:13.048 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:31:12 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7[55156]: 2024-09-06T14:31:12.721+0000 7fa8a1380640 -1 osd.7 64 heartbeat_check: no reply from 172.21.15.142:6807 osd.1 since back 2024-09-06T14:30:31.633604+0000 front 2024-09-06T14:30:31.633572+0000 (oldest deadline 2024-09-06T14:30:56.933451+0000) 2024-09-06T14:31:13.048 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:31:12 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7[55156]: 2024-09-06T14:31:12.721+0000 7fa8a1380640 -1 osd.7 64 heartbeat_check: no reply from 172.21.15.142:6811 osd.2 since back 2024-09-06T14:30:42.134879+0000 front 2024-09-06T14:30:42.134961+0000 (oldest deadline 2024-09-06T14:31:06.834776+0000) 2024-09-06T14:31:13.049 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:31:12 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7[55156]: 2024-09-06T14:31:12.721+0000 7fa8a1380640 -1 osd.7 64 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24421.0:364 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64) 2024-09-06T14:31:13.548 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 06 14:31:13 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-6[49599]: 2024-09-06T14:31:13.170+0000 7fef2ba64640 -1 osd.6 64 heartbeat_check: no reply from 172.21.15.142:6803 osd.0 since back 2024-09-06T14:30:25.822692+0000 front 2024-09-06T14:30:25.822628+0000 (oldest deadline 2024-09-06T14:30:51.122413+0000) 2024-09-06T14:31:13.548 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 06 14:31:13 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-6[49599]: 2024-09-06T14:31:13.170+0000 7fef2ba64640 -1 osd.6 64 heartbeat_check: no reply from 172.21.15.142:6807 osd.1 since back 2024-09-06T14:30:31.122845+0000 front 2024-09-06T14:30:31.122767+0000 (oldest deadline 2024-09-06T14:30:57.022798+0000) 2024-09-06T14:31:13.548 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 06 14:31:13 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-6[49599]: 2024-09-06T14:31:13.170+0000 7fef2ba64640 -1 osd.6 64 heartbeat_check: no reply from 172.21.15.142:6811 osd.2 since back 2024-09-06T14:30:42.923955+0000 front 2024-09-06T14:30:42.923941+0000 (oldest deadline 2024-09-06T14:31:06.423830+0000) 2024-09-06T14:31:14.048 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:31:13 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7[55156]: 2024-09-06T14:31:13.685+0000 7fa8a1380640 -1 osd.7 64 heartbeat_check: no reply from 172.21.15.142:6803 osd.0 since back 2024-09-06T14:30:25.832410+0000 front 2024-09-06T14:30:25.832610+0000 (oldest deadline 2024-09-06T14:30:49.332618+0000) 2024-09-06T14:31:14.048 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:31:13 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7[55156]: 2024-09-06T14:31:13.685+0000 7fa8a1380640 -1 osd.7 64 heartbeat_check: no reply from 172.21.15.142:6807 osd.1 since back 2024-09-06T14:30:31.633604+0000 front 2024-09-06T14:30:31.633572+0000 (oldest deadline 2024-09-06T14:30:56.933451+0000) 2024-09-06T14:31:14.048 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:31:13 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7[55156]: 2024-09-06T14:31:13.685+0000 7fa8a1380640 -1 osd.7 64 heartbeat_check: no reply from 172.21.15.142:6811 osd.2 since back 2024-09-06T14:30:42.134879+0000 front 2024-09-06T14:30:42.134961+0000 (oldest deadline 2024-09-06T14:31:06.834776+0000) 2024-09-06T14:31:14.048 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:31:13 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7[55156]: 2024-09-06T14:31:13.685+0000 7fa8a1380640 -1 osd.7 64 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24421.0:364 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64) 2024-09-06T14:31:14.548 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 06 14:31:14 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-6[49599]: 2024-09-06T14:31:14.130+0000 7fef2ba64640 -1 osd.6 64 heartbeat_check: no reply from 172.21.15.142:6803 osd.0 since back 2024-09-06T14:30:25.822692+0000 front 2024-09-06T14:30:25.822628+0000 (oldest deadline 2024-09-06T14:30:51.122413+0000) 2024-09-06T14:31:14.548 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 06 14:31:14 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-6[49599]: 2024-09-06T14:31:14.130+0000 7fef2ba64640 -1 osd.6 64 heartbeat_check: no reply from 172.21.15.142:6807 osd.1 since back 2024-09-06T14:30:31.122845+0000 front 2024-09-06T14:30:31.122767+0000 (oldest deadline 2024-09-06T14:30:57.022798+0000) 2024-09-06T14:31:14.548 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 06 14:31:14 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-6[49599]: 2024-09-06T14:31:14.130+0000 7fef2ba64640 -1 osd.6 64 heartbeat_check: no reply from 172.21.15.142:6811 osd.2 since back 2024-09-06T14:30:42.923955+0000 front 2024-09-06T14:30:42.923941+0000 (oldest deadline 2024-09-06T14:31:06.423830+0000) 2024-09-06T14:31:15.048 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:31:14 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7[55156]: 2024-09-06T14:31:14.713+0000 7fa8a1380640 -1 osd.7 64 heartbeat_check: no reply from 172.21.15.142:6803 osd.0 since back 2024-09-06T14:30:25.832410+0000 front 2024-09-06T14:30:25.832610+0000 (oldest deadline 2024-09-06T14:30:49.332618+0000) 2024-09-06T14:31:15.048 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:31:14 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7[55156]: 2024-09-06T14:31:14.713+0000 7fa8a1380640 -1 osd.7 64 heartbeat_check: no reply from 172.21.15.142:6807 osd.1 since back 2024-09-06T14:30:31.633604+0000 front 2024-09-06T14:30:31.633572+0000 (oldest deadline 2024-09-06T14:30:56.933451+0000) 2024-09-06T14:31:15.048 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:31:14 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7[55156]: 2024-09-06T14:31:14.713+0000 7fa8a1380640 -1 osd.7 64 heartbeat_check: no reply from 172.21.15.142:6811 osd.2 since back 2024-09-06T14:30:42.134879+0000 front 2024-09-06T14:30:42.134961+0000 (oldest deadline 2024-09-06T14:31:06.834776+0000) 2024-09-06T14:31:15.048 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:31:14 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7[55156]: 2024-09-06T14:31:14.713+0000 7fa8a1380640 -1 osd.7 64 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24421.0:364 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64) 2024-09-06T14:31:15.548 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 06 14:31:15 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-6[49599]: 2024-09-06T14:31:15.166+0000 7fef2ba64640 -1 osd.6 64 heartbeat_check: no reply from 172.21.15.142:6803 osd.0 since back 2024-09-06T14:30:25.822692+0000 front 2024-09-06T14:30:25.822628+0000 (oldest deadline 2024-09-06T14:30:51.122413+0000) 2024-09-06T14:31:15.548 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 06 14:31:15 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-6[49599]: 2024-09-06T14:31:15.166+0000 7fef2ba64640 -1 osd.6 64 heartbeat_check: no reply from 172.21.15.142:6807 osd.1 since back 2024-09-06T14:30:31.122845+0000 front 2024-09-06T14:30:31.122767+0000 (oldest deadline 2024-09-06T14:30:57.022798+0000) 2024-09-06T14:31:15.548 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 06 14:31:15 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-6[49599]: 2024-09-06T14:31:15.166+0000 7fef2ba64640 -1 osd.6 64 heartbeat_check: no reply from 172.21.15.142:6811 osd.2 since back 2024-09-06T14:30:42.923955+0000 front 2024-09-06T14:30:42.923941+0000 (oldest deadline 2024-09-06T14:31:06.423830+0000) 2024-09-06T14:31:16.048 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:31:15 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7[55156]: 2024-09-06T14:31:15.731+0000 7fa8a1380640 -1 osd.7 64 heartbeat_check: no reply from 172.21.15.142:6803 osd.0 since back 2024-09-06T14:30:25.832410+0000 front 2024-09-06T14:30:25.832610+0000 (oldest deadline 2024-09-06T14:30:49.332618+0000) 2024-09-06T14:31:16.048 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:31:15 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7[55156]: 2024-09-06T14:31:15.731+0000 7fa8a1380640 -1 osd.7 64 heartbeat_check: no reply from 172.21.15.142:6807 osd.1 since back 2024-09-06T14:30:31.633604+0000 front 2024-09-06T14:30:31.633572+0000 (oldest deadline 2024-09-06T14:30:56.933451+0000) 2024-09-06T14:31:16.048 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:31:15 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7[55156]: 2024-09-06T14:31:15.731+0000 7fa8a1380640 -1 osd.7 64 heartbeat_check: no reply from 172.21.15.142:6811 osd.2 since back 2024-09-06T14:30:42.134879+0000 front 2024-09-06T14:30:42.134961+0000 (oldest deadline 2024-09-06T14:31:06.834776+0000) 2024-09-06T14:31:16.049 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:31:15 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7[55156]: 2024-09-06T14:31:15.732+0000 7fa8a1380640 -1 osd.7 64 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24421.0:364 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64) 2024-09-06T14:31:16.432 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 06 14:31:16 smithi162 podman[66519]: 2024-09-06 14:31:16.162160788 +0000 UTC m=+5.491298619 container died b5da2fb81a012d9faa25ae4580cda66ed2451ee8797cd6dcb375fccc00ce9df2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c86edd48571ab22373283621eb9feebce9df9c291d4164cfc4df1a09493e1f30, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-6, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, RELEASE=wip-athakkar-testing-2024-09-04-1837-8b38c33) 2024-09-06T14:31:16.750 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 06 14:31:16 smithi162 podman[66519]: 2024-09-06 14:31:16.533407633 +0000 UTC m=+5.862545453 container cleanup b5da2fb81a012d9faa25ae4580cda66ed2451ee8797cd6dcb375fccc00ce9df2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c86edd48571ab22373283621eb9feebce9df9c291d4164cfc4df1a09493e1f30, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-6, CEPH_POINT_RELEASE=, RELEASE=wip-athakkar-testing-2024-09-04-1837-8b38c33, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD) 2024-09-06T14:31:16.750 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 06 14:31:16 smithi162 bash[66519]: ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-6 2024-09-06T14:31:17.048 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 06 14:31:16 smithi162 podman[66534]: 2024-09-06 14:31:16.835245593 +0000 UTC m=+0.672413805 container remove b5da2fb81a012d9faa25ae4580cda66ed2451ee8797cd6dcb375fccc00ce9df2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c86edd48571ab22373283621eb9feebce9df9c291d4164cfc4df1a09493e1f30, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-6, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, RELEASE=wip-athakkar-testing-2024-09-04-1837-8b38c33) 2024-09-06T14:31:17.048 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:31:16 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7[55156]: 2024-09-06T14:31:16.752+0000 7fa8a1380640 -1 osd.7 64 heartbeat_check: no reply from 172.21.15.142:6803 osd.0 since back 2024-09-06T14:30:25.832410+0000 front 2024-09-06T14:30:25.832610+0000 (oldest deadline 2024-09-06T14:30:49.332618+0000) 2024-09-06T14:31:17.048 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:31:16 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7[55156]: 2024-09-06T14:31:16.752+0000 7fa8a1380640 -1 osd.7 64 heartbeat_check: no reply from 172.21.15.142:6807 osd.1 since back 2024-09-06T14:30:31.633604+0000 front 2024-09-06T14:30:31.633572+0000 (oldest deadline 2024-09-06T14:30:56.933451+0000) 2024-09-06T14:31:17.048 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:31:16 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7[55156]: 2024-09-06T14:31:16.752+0000 7fa8a1380640 -1 osd.7 64 heartbeat_check: no reply from 172.21.15.142:6811 osd.2 since back 2024-09-06T14:30:42.134879+0000 front 2024-09-06T14:30:42.134961+0000 (oldest deadline 2024-09-06T14:31:06.834776+0000) 2024-09-06T14:31:17.048 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:31:16 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7[55156]: 2024-09-06T14:31:16.752+0000 7fa8a1380640 -1 osd.7 64 heartbeat_check: no reply from 172.21.15.142:6815 osd.3 since back 2024-09-06T14:30:51.435843+0000 front 2024-09-06T14:30:51.435902+0000 (oldest deadline 2024-09-06T14:31:16.135795+0000) 2024-09-06T14:31:17.048 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:31:16 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7[55156]: 2024-09-06T14:31:16.752+0000 7fa8a1380640 -1 osd.7 64 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24421.0:364 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64) 2024-09-06T14:31:17.468 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 06 14:31:17 smithi162 podman[66625]: 2024-09-06 14:31:17.048051433 +0000 UTC m=+0.015945240 image pull 1c45aca0610a312233aa2613f7b416b2884ae14779f1297041485ab28f1c8110 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c86edd48571ab22373283621eb9feebce9df9c291d4164cfc4df1a09493e1f30 2024-09-06T14:31:17.469 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 06 14:31:17 smithi162 podman[66625]: 2024-09-06 14:31:17.17775558 +0000 UTC m=+0.145649449 container create 6a1389eedfa53bd2e2d35776d7dd0e9f795c0ba08c83a747c413b12c2f7385b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c86edd48571ab22373283621eb9feebce9df9c291d4164cfc4df1a09493e1f30, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-athakkar-testing-2024-09-04-1837-8b38c33, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2) 2024-09-06T14:31:17.469 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 06 14:31:17 smithi162 podman[66625]: 2024-09-06 14:31:17.380095104 +0000 UTC m=+0.347988911 container init 6a1389eedfa53bd2e2d35776d7dd0e9f795c0ba08c83a747c413b12c2f7385b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c86edd48571ab22373283621eb9feebce9df9c291d4164cfc4df1a09493e1f30, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-athakkar-testing-2024-09-04-1837-8b38c33, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0) 2024-09-06T14:31:17.469 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 06 14:31:17 smithi162 podman[66625]: 2024-09-06 14:31:17.383581782 +0000 UTC m=+0.351475590 container start 6a1389eedfa53bd2e2d35776d7dd0e9f795c0ba08c83a747c413b12c2f7385b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c86edd48571ab22373283621eb9feebce9df9c291d4164cfc4df1a09493e1f30, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-6-deactivate, RELEASE=wip-athakkar-testing-2024-09-04-1837-8b38c33, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-06T14:31:17.469 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 06 14:31:17 smithi162 podman[66625]: 2024-09-06 14:31:17.471785097 +0000 UTC m=+0.439678907 container attach 6a1389eedfa53bd2e2d35776d7dd0e9f795c0ba08c83a747c413b12c2f7385b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c86edd48571ab22373283621eb9feebce9df9c291d4164cfc4df1a09493e1f30, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-6-deactivate, RELEASE=wip-athakkar-testing-2024-09-04-1837-8b38c33, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS) 2024-09-06T14:31:17.782 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 06 14:31:17 smithi162 podman[66625]: 2024-09-06 14:31:17.573993877 +0000 UTC m=+0.541887694 container died 6a1389eedfa53bd2e2d35776d7dd0e9f795c0ba08c83a747c413b12c2f7385b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c86edd48571ab22373283621eb9feebce9df9c291d4164cfc4df1a09493e1f30, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-6-deactivate, io.buildah.version=1.37.2, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-athakkar-testing-2024-09-04-1837-8b38c33, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-09-06T14:31:18.048 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:31:17 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7[55156]: 2024-09-06T14:31:17.785+0000 7fa8a1380640 -1 osd.7 64 heartbeat_check: no reply from 172.21.15.142:6803 osd.0 since back 2024-09-06T14:30:25.832410+0000 front 2024-09-06T14:30:25.832610+0000 (oldest deadline 2024-09-06T14:30:49.332618+0000) 2024-09-06T14:31:18.048 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:31:17 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7[55156]: 2024-09-06T14:31:17.785+0000 7fa8a1380640 -1 osd.7 64 heartbeat_check: no reply from 172.21.15.142:6807 osd.1 since back 2024-09-06T14:30:31.633604+0000 front 2024-09-06T14:30:31.633572+0000 (oldest deadline 2024-09-06T14:30:56.933451+0000) 2024-09-06T14:31:18.048 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:31:17 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7[55156]: 2024-09-06T14:31:17.785+0000 7fa8a1380640 -1 osd.7 64 heartbeat_check: no reply from 172.21.15.142:6811 osd.2 since back 2024-09-06T14:30:42.134879+0000 front 2024-09-06T14:30:42.134961+0000 (oldest deadline 2024-09-06T14:31:06.834776+0000) 2024-09-06T14:31:18.048 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:31:17 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7[55156]: 2024-09-06T14:31:17.785+0000 7fa8a1380640 -1 osd.7 64 heartbeat_check: no reply from 172.21.15.142:6815 osd.3 since back 2024-09-06T14:30:51.435843+0000 front 2024-09-06T14:30:51.435902+0000 (oldest deadline 2024-09-06T14:31:16.135795+0000) 2024-09-06T14:31:18.048 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:31:17 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7[55156]: 2024-09-06T14:31:17.785+0000 7fa8a1380640 -1 osd.7 64 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24421.0:364 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64) 2024-09-06T14:31:18.175 DEBUG:teuthology.orchestra.run.smithi162:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ce32462e-6c5a-11ef-bce4-c7b262605968@osd.6.service' 2024-09-06T14:31:18.216 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-06T14:31:18.216 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-09-06T14:31:18.216 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-09-06T14:31:18.216 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ceph-ce32462e-6c5a-11ef-bce4-c7b262605968@osd.7 2024-09-06T14:31:18.542 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:31:18 smithi162 systemd[1]: Stopping Ceph osd.7 for ce32462e-6c5a-11ef-bce4-c7b262605968... 2024-09-06T14:31:18.542 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:31:18 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7[55156]: 2024-09-06T14:31:18.544+0000 7fa8a8d73640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-06T14:31:18.542 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:31:18 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7[55156]: 2024-09-06T14:31:18.544+0000 7fa8a8d73640 -1 osd.7 64 *** Got signal Terminated *** 2024-09-06T14:31:18.543 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:31:18 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7[55156]: 2024-09-06T14:31:18.544+0000 7fa8a8d73640 -1 osd.7 64 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-06T14:31:18.798 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:31:18 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7[55156]: 2024-09-06T14:31:18.769+0000 7fa8a1380640 -1 osd.7 64 heartbeat_check: no reply from 172.21.15.142:6803 osd.0 since back 2024-09-06T14:30:25.832410+0000 front 2024-09-06T14:30:25.832610+0000 (oldest deadline 2024-09-06T14:30:49.332618+0000) 2024-09-06T14:31:18.798 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:31:18 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7[55156]: 2024-09-06T14:31:18.769+0000 7fa8a1380640 -1 osd.7 64 heartbeat_check: no reply from 172.21.15.142:6807 osd.1 since back 2024-09-06T14:30:31.633604+0000 front 2024-09-06T14:30:31.633572+0000 (oldest deadline 2024-09-06T14:30:56.933451+0000) 2024-09-06T14:31:18.798 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:31:18 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7[55156]: 2024-09-06T14:31:18.769+0000 7fa8a1380640 -1 osd.7 64 heartbeat_check: no reply from 172.21.15.142:6811 osd.2 since back 2024-09-06T14:30:42.134879+0000 front 2024-09-06T14:30:42.134961+0000 (oldest deadline 2024-09-06T14:31:06.834776+0000) 2024-09-06T14:31:18.798 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:31:18 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7[55156]: 2024-09-06T14:31:18.769+0000 7fa8a1380640 -1 osd.7 64 heartbeat_check: no reply from 172.21.15.142:6815 osd.3 since back 2024-09-06T14:30:51.435843+0000 front 2024-09-06T14:30:51.435902+0000 (oldest deadline 2024-09-06T14:31:16.135795+0000) 2024-09-06T14:31:18.798 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:31:18 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7[55156]: 2024-09-06T14:31:18.769+0000 7fa8a1380640 -1 osd.7 64 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24421.0:364 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64) 2024-09-06T14:31:20.048 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:31:19 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7[55156]: 2024-09-06T14:31:19.784+0000 7fa8a1380640 -1 osd.7 64 heartbeat_check: no reply from 172.21.15.142:6803 osd.0 since back 2024-09-06T14:30:25.832410+0000 front 2024-09-06T14:30:25.832610+0000 (oldest deadline 2024-09-06T14:30:49.332618+0000) 2024-09-06T14:31:20.048 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:31:19 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7[55156]: 2024-09-06T14:31:19.784+0000 7fa8a1380640 -1 osd.7 64 heartbeat_check: no reply from 172.21.15.142:6807 osd.1 since back 2024-09-06T14:30:31.633604+0000 front 2024-09-06T14:30:31.633572+0000 (oldest deadline 2024-09-06T14:30:56.933451+0000) 2024-09-06T14:31:20.048 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:31:19 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7[55156]: 2024-09-06T14:31:19.784+0000 7fa8a1380640 -1 osd.7 64 heartbeat_check: no reply from 172.21.15.142:6811 osd.2 since back 2024-09-06T14:30:42.134879+0000 front 2024-09-06T14:30:42.134961+0000 (oldest deadline 2024-09-06T14:31:06.834776+0000) 2024-09-06T14:31:20.048 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:31:19 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7[55156]: 2024-09-06T14:31:19.784+0000 7fa8a1380640 -1 osd.7 64 heartbeat_check: no reply from 172.21.15.142:6815 osd.3 since back 2024-09-06T14:30:51.435843+0000 front 2024-09-06T14:30:51.435902+0000 (oldest deadline 2024-09-06T14:31:16.135795+0000) 2024-09-06T14:31:20.048 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:31:19 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7[55156]: 2024-09-06T14:31:19.784+0000 7fa8a1380640 -1 osd.7 64 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24421.0:364 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64) 2024-09-06T14:31:21.271 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:31:20 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7[55156]: 2024-09-06T14:31:20.819+0000 7fa8a1380640 -1 osd.7 64 heartbeat_check: no reply from 172.21.15.142:6803 osd.0 since back 2024-09-06T14:30:25.832410+0000 front 2024-09-06T14:30:25.832610+0000 (oldest deadline 2024-09-06T14:30:49.332618+0000) 2024-09-06T14:31:21.271 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:31:20 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7[55156]: 2024-09-06T14:31:20.819+0000 7fa8a1380640 -1 osd.7 64 heartbeat_check: no reply from 172.21.15.142:6807 osd.1 since back 2024-09-06T14:30:31.633604+0000 front 2024-09-06T14:30:31.633572+0000 (oldest deadline 2024-09-06T14:30:56.933451+0000) 2024-09-06T14:31:21.271 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:31:20 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7[55156]: 2024-09-06T14:31:20.819+0000 7fa8a1380640 -1 osd.7 64 heartbeat_check: no reply from 172.21.15.142:6811 osd.2 since back 2024-09-06T14:30:42.134879+0000 front 2024-09-06T14:30:42.134961+0000 (oldest deadline 2024-09-06T14:31:06.834776+0000) 2024-09-06T14:31:21.271 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:31:20 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7[55156]: 2024-09-06T14:31:20.819+0000 7fa8a1380640 -1 osd.7 64 heartbeat_check: no reply from 172.21.15.142:6815 osd.3 since back 2024-09-06T14:30:51.435843+0000 front 2024-09-06T14:30:51.435902+0000 (oldest deadline 2024-09-06T14:31:16.135795+0000) 2024-09-06T14:31:21.271 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:31:20 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7[55156]: 2024-09-06T14:31:20.819+0000 7fa8a1380640 -1 osd.7 64 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24421.0:364 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64) 2024-09-06T14:31:22.048 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:31:21 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7[55156]: 2024-09-06T14:31:21.772+0000 7fa8a1380640 -1 osd.7 64 heartbeat_check: no reply from 172.21.15.142:6803 osd.0 since back 2024-09-06T14:30:25.832410+0000 front 2024-09-06T14:30:25.832610+0000 (oldest deadline 2024-09-06T14:30:49.332618+0000) 2024-09-06T14:31:22.048 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:31:21 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7[55156]: 2024-09-06T14:31:21.772+0000 7fa8a1380640 -1 osd.7 64 heartbeat_check: no reply from 172.21.15.142:6807 osd.1 since back 2024-09-06T14:30:31.633604+0000 front 2024-09-06T14:30:31.633572+0000 (oldest deadline 2024-09-06T14:30:56.933451+0000) 2024-09-06T14:31:22.048 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:31:21 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7[55156]: 2024-09-06T14:31:21.772+0000 7fa8a1380640 -1 osd.7 64 heartbeat_check: no reply from 172.21.15.142:6811 osd.2 since back 2024-09-06T14:30:42.134879+0000 front 2024-09-06T14:30:42.134961+0000 (oldest deadline 2024-09-06T14:31:06.834776+0000) 2024-09-06T14:31:22.048 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:31:21 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7[55156]: 2024-09-06T14:31:21.772+0000 7fa8a1380640 -1 osd.7 64 heartbeat_check: no reply from 172.21.15.142:6815 osd.3 since back 2024-09-06T14:30:51.435843+0000 front 2024-09-06T14:30:51.435902+0000 (oldest deadline 2024-09-06T14:31:16.135795+0000) 2024-09-06T14:31:23.048 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:31:22 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7[55156]: 2024-09-06T14:31:22.791+0000 7fa8a1380640 -1 osd.7 64 heartbeat_check: no reply from 172.21.15.142:6803 osd.0 since back 2024-09-06T14:30:25.832410+0000 front 2024-09-06T14:30:25.832610+0000 (oldest deadline 2024-09-06T14:30:49.332618+0000) 2024-09-06T14:31:23.048 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:31:22 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7[55156]: 2024-09-06T14:31:22.791+0000 7fa8a1380640 -1 osd.7 64 heartbeat_check: no reply from 172.21.15.142:6807 osd.1 since back 2024-09-06T14:30:31.633604+0000 front 2024-09-06T14:30:31.633572+0000 (oldest deadline 2024-09-06T14:30:56.933451+0000) 2024-09-06T14:31:23.048 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:31:22 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7[55156]: 2024-09-06T14:31:22.791+0000 7fa8a1380640 -1 osd.7 64 heartbeat_check: no reply from 172.21.15.142:6811 osd.2 since back 2024-09-06T14:30:42.134879+0000 front 2024-09-06T14:30:42.134961+0000 (oldest deadline 2024-09-06T14:31:06.834776+0000) 2024-09-06T14:31:23.048 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:31:22 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7[55156]: 2024-09-06T14:31:22.791+0000 7fa8a1380640 -1 osd.7 64 heartbeat_check: no reply from 172.21.15.142:6815 osd.3 since back 2024-09-06T14:30:51.435843+0000 front 2024-09-06T14:30:51.435902+0000 (oldest deadline 2024-09-06T14:31:16.135795+0000) 2024-09-06T14:31:23.048 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:31:22 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7[55156]: 2024-09-06T14:31:22.791+0000 7fa8a1380640 -1 osd.7 64 heartbeat_check: no reply from 172.21.15.162:6802 osd.4 since back 2024-09-06T14:30:58.937236+0000 front 2024-09-06T14:30:58.937325+0000 (oldest deadline 2024-09-06T14:31:21.837189+0000) 2024-09-06T14:31:24.141 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:31:23 smithi162 podman[66728]: 2024-09-06 14:31:23.854661139 +0000 UTC m=+5.407205789 container died bad219aeca1b08628ea687847e44dfded2b39a9af80672872fb9b210475a7db0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c86edd48571ab22373283621eb9feebce9df9c291d4164cfc4df1a09493e1f30, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-athakkar-testing-2024-09-04-1837-8b38c33, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0) 2024-09-06T14:31:24.141 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:31:24 smithi162 podman[66728]: 2024-09-06 14:31:24.144529776 +0000 UTC m=+5.697074415 container cleanup bad219aeca1b08628ea687847e44dfded2b39a9af80672872fb9b210475a7db0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c86edd48571ab22373283621eb9feebce9df9c291d4164cfc4df1a09493e1f30, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7, GIT_BRANCH=HEAD, RELEASE=wip-athakkar-testing-2024-09-04-1837-8b38c33, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-09-06T14:31:24.548 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:31:24 smithi162 bash[66728]: ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7 2024-09-06T14:31:24.548 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:31:24 smithi162 podman[66754]: 2024-09-06 14:31:24.387254621 +0000 UTC m=+0.528918758 container remove bad219aeca1b08628ea687847e44dfded2b39a9af80672872fb9b210475a7db0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c86edd48571ab22373283621eb9feebce9df9c291d4164cfc4df1a09493e1f30, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-athakkar-testing-2024-09-04-1837-8b38c33, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-09-06T14:31:24.953 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:31:24 smithi162 podman[66844]: 2024-09-06 14:31:24.609543289 +0000 UTC m=+0.016104382 image pull 1c45aca0610a312233aa2613f7b416b2884ae14779f1297041485ab28f1c8110 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c86edd48571ab22373283621eb9feebce9df9c291d4164cfc4df1a09493e1f30 2024-09-06T14:31:24.954 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:31:24 smithi162 podman[66844]: 2024-09-06 14:31:24.713198604 +0000 UTC m=+0.119759723 container create 4f66e3c1d5efa62be32bf8befebe8c2fb7f1702e739480092113467d185a553c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c86edd48571ab22373283621eb9feebce9df9c291d4164cfc4df1a09493e1f30, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7-deactivate, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-athakkar-testing-2024-09-04-1837-8b38c33, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-06T14:31:25.298 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:31:24 smithi162 podman[66844]: 2024-09-06 14:31:24.957181104 +0000 UTC m=+0.363742178 container init 4f66e3c1d5efa62be32bf8befebe8c2fb7f1702e739480092113467d185a553c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c86edd48571ab22373283621eb9feebce9df9c291d4164cfc4df1a09493e1f30, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-athakkar-testing-2024-09-04-1837-8b38c33, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2) 2024-09-06T14:31:25.298 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:31:24 smithi162 podman[66844]: 2024-09-06 14:31:24.960534616 +0000 UTC m=+0.367095700 container start 4f66e3c1d5efa62be32bf8befebe8c2fb7f1702e739480092113467d185a553c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c86edd48571ab22373283621eb9feebce9df9c291d4164cfc4df1a09493e1f30, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7-deactivate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-athakkar-testing-2024-09-04-1837-8b38c33) 2024-09-06T14:31:25.298 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:31:25 smithi162 podman[66844]: 2024-09-06 14:31:25.04060396 +0000 UTC m=+0.447165044 container attach 4f66e3c1d5efa62be32bf8befebe8c2fb7f1702e739480092113467d185a553c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c86edd48571ab22373283621eb9feebce9df9c291d4164cfc4df1a09493e1f30, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7-deactivate, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-athakkar-testing-2024-09-04-1837-8b38c33, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux ) 2024-09-06T14:31:25.298 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:31:25 smithi162 podman[66844]: 2024-09-06 14:31:25.156173614 +0000 UTC m=+0.562734697 container died 4f66e3c1d5efa62be32bf8befebe8c2fb7f1702e739480092113467d185a553c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c86edd48571ab22373283621eb9feebce9df9c291d4164cfc4df1a09493e1f30, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7-deactivate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, RELEASE=wip-athakkar-testing-2024-09-04-1837-8b38c33, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-06T14:31:25.677 DEBUG:teuthology.orchestra.run.smithi162:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ce32462e-6c5a-11ef-bce4-c7b262605968@osd.7.service' 2024-09-06T14:31:25.702 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:31:25 smithi162 podman[66867]: 2024-09-06 14:31:25.652507451 +0000 UTC m=+0.490209602 container remove 4f66e3c1d5efa62be32bf8befebe8c2fb7f1702e739480092113467d185a553c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c86edd48571ab22373283621eb9feebce9df9c291d4164cfc4df1a09493e1f30, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-athakkar-testing-2024-09-04-1837-8b38c33, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-06T14:31:25.703 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:31:25 smithi162 systemd[1]: ceph-ce32462e-6c5a-11ef-bce4-c7b262605968@osd.7.service: Deactivated successfully. 2024-09-06T14:31:25.703 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:31:25 smithi162 systemd[1]: Stopped Ceph osd.7 for ce32462e-6c5a-11ef-bce4-c7b262605968. 2024-09-06T14:31:25.703 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 06 14:31:25 smithi162 systemd[1]: ceph-ce32462e-6c5a-11ef-bce4-c7b262605968@osd.7.service: Consumed 4.464s CPU time. 2024-09-06T14:31:25.719 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-06T14:31:25.719 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-09-06T14:31:25.719 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopping rgw.foo.a... 2024-09-06T14:31:25.720 DEBUG:teuthology.orchestra.run.smithi142:> sudo systemctl stop ceph-ce32462e-6c5a-11ef-bce4-c7b262605968@rgw.foo.a 2024-09-06T14:31:26.025 INFO:journalctl@ceph.rgw.foo.a.smithi142.stdout:Sep 06 14:31:25 smithi142 systemd[1]: Stopping Ceph rgw.foo.a for ce32462e-6c5a-11ef-bce4-c7b262605968... 2024-09-06T14:31:26.280 INFO:journalctl@ceph.rgw.foo.a.smithi142.stdout:Sep 06 14:31:26 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-rgw-foo-a[63255]: 2024-09-06T14:31:26.027+0000 7f173a28a640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/radosgw -n client.rgw.foo.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-06T14:31:26.281 INFO:journalctl@ceph.rgw.foo.a.smithi142.stdout:Sep 06 14:31:26 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-rgw-foo-a[63255]: 2024-09-06T14:31:26.027+0000 7f173daf6880 -1 shutting down 2024-09-06T14:31:36.342 INFO:journalctl@ceph.rgw.foo.a.smithi142.stdout:Sep 06 14:31:36 smithi142 bash[81978]: time="2024-09-06T14:31:36Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-rgw-foo-a in 10 seconds, resorting to SIGKILL" 2024-09-06T14:31:36.342 INFO:journalctl@ceph.rgw.foo.a.smithi142.stdout:Sep 06 14:31:36 smithi142 podman[81978]: 2024-09-06 14:31:36.060680257 +0000 UTC m=+10.141631091 container died 47d61cae342f69fa2cb6b26d9e0422c4fadd727043e2c381234c1c3e711b6b0c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c86edd48571ab22373283621eb9feebce9df9c291d4164cfc4df1a09493e1f30, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-rgw-foo-a, RELEASE=wip-athakkar-testing-2024-09-04-1837-8b38c33, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-06T14:31:36.709 INFO:journalctl@ceph.rgw.foo.a.smithi142.stdout:Sep 06 14:31:36 smithi142 podman[81978]: 2024-09-06 14:31:36.543519277 +0000 UTC m=+10.624470093 container cleanup 47d61cae342f69fa2cb6b26d9e0422c4fadd727043e2c381234c1c3e711b6b0c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c86edd48571ab22373283621eb9feebce9df9c291d4164cfc4df1a09493e1f30, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-rgw-foo-a, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-athakkar-testing-2024-09-04-1837-8b38c33, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-09-06T14:31:36.709 INFO:journalctl@ceph.rgw.foo.a.smithi142.stdout:Sep 06 14:31:36 smithi142 bash[81978]: ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-rgw-foo-a 2024-09-06T14:31:36.980 DEBUG:teuthology.orchestra.run.smithi142:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ce32462e-6c5a-11ef-bce4-c7b262605968@rgw.foo.a.service' 2024-09-06T14:31:37.002 INFO:journalctl@ceph.rgw.foo.a.smithi142.stdout:Sep 06 14:31:36 smithi142 podman[82003]: 2024-09-06 14:31:36.819789271 +0000 UTC m=+0.757865669 container remove 47d61cae342f69fa2cb6b26d9e0422c4fadd727043e2c381234c1c3e711b6b0c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c86edd48571ab22373283621eb9feebce9df9c291d4164cfc4df1a09493e1f30, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-rgw-foo-a, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-athakkar-testing-2024-09-04-1837-8b38c33, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-09-06T14:31:37.002 INFO:journalctl@ceph.rgw.foo.a.smithi142.stdout:Sep 06 14:31:36 smithi142 systemd[1]: ceph-ce32462e-6c5a-11ef-bce4-c7b262605968@rgw.foo.a.service: Main process exited, code=exited, status=137/n/a 2024-09-06T14:31:37.002 INFO:journalctl@ceph.rgw.foo.a.smithi142.stdout:Sep 06 14:31:36 smithi142 systemd[1]: ceph-ce32462e-6c5a-11ef-bce4-c7b262605968@rgw.foo.a.service: Failed with result 'exit-code'. 2024-09-06T14:31:37.002 INFO:journalctl@ceph.rgw.foo.a.smithi142.stdout:Sep 06 14:31:36 smithi142 systemd[1]: Stopped Ceph rgw.foo.a for ce32462e-6c5a-11ef-bce4-c7b262605968. 2024-09-06T14:31:37.003 INFO:journalctl@ceph.rgw.foo.a.smithi142.stdout:Sep 06 14:31:36 smithi142 systemd[1]: ceph-ce32462e-6c5a-11ef-bce4-c7b262605968@rgw.foo.a.service: Consumed 2.027s CPU time. 2024-09-06T14:31:37.016 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-06T14:31:37.017 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopped rgw.foo.a 2024-09-06T14:31:37.017 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-09-06T14:31:37.017 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ceph-ce32462e-6c5a-11ef-bce4-c7b262605968@prometheus.a 2024-09-06T14:31:37.298 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 06 14:31:37 smithi162 systemd[1]: Stopping Ceph prometheus.a for ce32462e-6c5a-11ef-bce4-c7b262605968... 2024-09-06T14:31:37.298 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 06 14:31:37 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-prometheus-a[64015]: ts=2024-09-06T14:31:37.295Z caller=main.go:964 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-06T14:31:37.298 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 06 14:31:37 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-prometheus-a[64015]: ts=2024-09-06T14:31:37.295Z caller=main.go:988 level=info msg="Stopping scrape discovery manager..." 2024-09-06T14:31:37.298 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 06 14:31:37 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-prometheus-a[64015]: ts=2024-09-06T14:31:37.295Z caller=main.go:1002 level=info msg="Stopping notify discovery manager..." 2024-09-06T14:31:37.299 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 06 14:31:37 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-prometheus-a[64015]: ts=2024-09-06T14:31:37.295Z caller=main.go:984 level=info msg="Scrape discovery manager stopped" 2024-09-06T14:31:37.299 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 06 14:31:37 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-prometheus-a[64015]: ts=2024-09-06T14:31:37.295Z caller=main.go:998 level=info msg="Notify discovery manager stopped" 2024-09-06T14:31:37.299 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 06 14:31:37 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-prometheus-a[64015]: ts=2024-09-06T14:31:37.295Z caller=manager.go:177 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-06T14:31:37.299 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 06 14:31:37 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-prometheus-a[64015]: ts=2024-09-06T14:31:37.295Z caller=manager.go:187 level=info component="rule manager" msg="Rule manager stopped" 2024-09-06T14:31:37.299 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 06 14:31:37 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-prometheus-a[64015]: ts=2024-09-06T14:31:37.295Z caller=main.go:1039 level=info msg="Stopping scrape manager..." 2024-09-06T14:31:37.299 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 06 14:31:37 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-prometheus-a[64015]: ts=2024-09-06T14:31:37.295Z caller=main.go:1031 level=info msg="Scrape manager stopped" 2024-09-06T14:31:37.299 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 06 14:31:37 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-prometheus-a[64015]: ts=2024-09-06T14:31:37.296Z caller=notifier.go:618 level=info component=notifier msg="Stopping notification manager..." 2024-09-06T14:31:37.299 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 06 14:31:37 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-prometheus-a[64015]: ts=2024-09-06T14:31:37.296Z caller=main.go:1261 level=info msg="Notifier manager stopped" 2024-09-06T14:31:37.299 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 06 14:31:37 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-prometheus-a[64015]: ts=2024-09-06T14:31:37.296Z caller=main.go:1273 level=info msg="See you next time!" 2024-09-06T14:31:37.560 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 06 14:31:37 smithi162 podman[66947]: 2024-09-06 14:31:37.305876258 +0000 UTC m=+0.087702792 container died f5d9f09518b211980b023ad3cd696941462f57fa427a957b5eea62b9e75ca41e (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-06T14:31:37.901 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 06 14:31:37 smithi162 podman[66947]: 2024-09-06 14:31:37.661838189 +0000 UTC m=+0.443664723 container cleanup f5d9f09518b211980b023ad3cd696941462f57fa427a957b5eea62b9e75ca41e (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-06T14:31:37.901 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 06 14:31:37 smithi162 bash[66947]: ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-prometheus-a 2024-09-06T14:31:38.033 DEBUG:teuthology.orchestra.run.smithi162:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ce32462e-6c5a-11ef-bce4-c7b262605968@prometheus.a.service' 2024-09-06T14:31:38.069 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-06T14:31:38.069 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-09-06T14:31:38.070 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm rm-cluster --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 --force --keep-logs 2024-09-06T14:31:38.271 INFO:teuthology.orchestra.run.smithi142.stdout:Deleting cluster with fsid: ce32462e-6c5a-11ef-bce4-c7b262605968 2024-09-06T14:31:40.154 INFO:journalctl@ceph.alertmanager.a.smithi142.stdout:Sep 06 14:31:39 smithi142 systemd[1]: Stopping Ceph alertmanager.a for ce32462e-6c5a-11ef-bce4-c7b262605968... 2024-09-06T14:31:40.154 INFO:journalctl@ceph.alertmanager.a.smithi142.stdout:Sep 06 14:31:40 smithi142 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-alertmanager-a[72766]: ts=2024-09-06T14:31:40.115Z caller=main.go:542 level=info msg="Received SIGTERM, exiting gracefully..." 2024-09-06T14:31:40.438 INFO:journalctl@ceph.alertmanager.a.smithi142.stdout:Sep 06 14:31:40 smithi142 podman[82483]: 2024-09-06 14:31:40.156867604 +0000 UTC m=+0.142513809 container died d4eed5ed6c6f478dc7588d07f643a91a333462ad630ac643956587fe5c7618e8 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-06T14:31:40.709 INFO:journalctl@ceph.alertmanager.a.smithi142.stdout:Sep 06 14:31:40 smithi142 podman[82483]: 2024-09-06 14:31:40.607470671 +0000 UTC m=+0.593116832 container cleanup d4eed5ed6c6f478dc7588d07f643a91a333462ad630ac643956587fe5c7618e8 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-06T14:31:40.709 INFO:journalctl@ceph.alertmanager.a.smithi142.stdout:Sep 06 14:31:40 smithi142 bash[82483]: ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-alertmanager-a 2024-09-06T14:31:41.161 INFO:journalctl@ceph.alertmanager.a.smithi142.stdout:Sep 06 14:31:40 smithi142 podman[82496]: 2024-09-06 14:31:40.809023893 +0000 UTC m=+0.642381186 container remove d4eed5ed6c6f478dc7588d07f643a91a333462ad630ac643956587fe5c7618e8 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-06T14:31:41.161 INFO:journalctl@ceph.alertmanager.a.smithi142.stdout:Sep 06 14:31:40 smithi142 podman[82496]: 2024-09-06 14:31:40.884429089 +0000 UTC m=+0.717786382 volume remove c31a8ef4c7cd4b820c2aba27fd61001f6f18f3acdd99e69289271034856ef7ca 2024-09-06T14:31:41.459 INFO:journalctl@ceph.node-exporter.a.smithi142.stdout:Sep 06 14:31:41 smithi142 systemd[1]: Stopping Ceph node-exporter.a for ce32462e-6c5a-11ef-bce4-c7b262605968... 2024-09-06T14:31:41.459 INFO:journalctl@ceph.alertmanager.a.smithi142.stdout:Sep 06 14:31:41 smithi142 systemd[1]: ceph-ce32462e-6c5a-11ef-bce4-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-09-06T14:31:41.460 INFO:journalctl@ceph.alertmanager.a.smithi142.stdout:Sep 06 14:31:41 smithi142 systemd[1]: Stopped Ceph alertmanager.a for ce32462e-6c5a-11ef-bce4-c7b262605968. 2024-09-06T14:31:41.460 INFO:journalctl@ceph.alertmanager.a.smithi142.stdout:Sep 06 14:31:41 smithi142 systemd[1]: ceph-ce32462e-6c5a-11ef-bce4-c7b262605968@alertmanager.a.service: Consumed 1.018s CPU time. 2024-09-06T14:31:41.884 INFO:journalctl@ceph.node-exporter.a.smithi142.stdout:Sep 06 14:31:41 smithi142 podman[82610]: 2024-09-06 14:31:41.63100067 +0000 UTC m=+0.107840397 container died ffc6b152297c8bf3e77095156e3b4daf087ef3537f23553b3dd13ca97a39f587 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-06T14:31:42.209 INFO:journalctl@ceph.node-exporter.a.smithi142.stdout:Sep 06 14:31:42 smithi142 podman[82610]: 2024-09-06 14:31:42.003215082 +0000 UTC m=+0.480054810 container cleanup ffc6b152297c8bf3e77095156e3b4daf087ef3537f23553b3dd13ca97a39f587 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-06T14:31:42.209 INFO:journalctl@ceph.node-exporter.a.smithi142.stdout:Sep 06 14:31:42 smithi142 bash[82610]: ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-a 2024-09-06T14:31:42.554 INFO:journalctl@ceph.node-exporter.a.smithi142.stdout:Sep 06 14:31:42 smithi142 podman[82624]: 2024-09-06 14:31:42.263111677 +0000 UTC m=+0.634579350 container remove ffc6b152297c8bf3e77095156e3b4daf087ef3537f23553b3dd13ca97a39f587 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-06T14:31:42.554 INFO:journalctl@ceph.node-exporter.a.smithi142.stdout:Sep 06 14:31:42 smithi142 systemd[1]: ceph-ce32462e-6c5a-11ef-bce4-c7b262605968@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-09-06T14:31:42.554 INFO:journalctl@ceph.node-exporter.a.smithi142.stdout:Sep 06 14:31:42 smithi142 systemd[1]: ceph-ce32462e-6c5a-11ef-bce4-c7b262605968@node-exporter.a.service: Failed with result 'exit-code'. 2024-09-06T14:31:42.554 INFO:journalctl@ceph.node-exporter.a.smithi142.stdout:Sep 06 14:31:42 smithi142 systemd[1]: Stopped Ceph node-exporter.a for ce32462e-6c5a-11ef-bce4-c7b262605968. 2024-09-06T14:31:42.554 INFO:journalctl@ceph.node-exporter.a.smithi142.stdout:Sep 06 14:31:42 smithi142 systemd[1]: ceph-ce32462e-6c5a-11ef-bce4-c7b262605968@node-exporter.a.service: Consumed 2.272s CPU time. 2024-09-06T14:31:43.155 DEBUG:teuthology.orchestra.run.smithi162:> sudo cephadm rm-cluster --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 --force --keep-logs 2024-09-06T14:31:43.358 INFO:teuthology.orchestra.run.smithi162.stdout:Deleting cluster with fsid: ce32462e-6c5a-11ef-bce4-c7b262605968 2024-09-06T14:31:45.048 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 06 14:31:44 smithi162 systemd[1]: Stopping Ceph iscsi.iscsi.a for ce32462e-6c5a-11ef-bce4-c7b262605968... 2024-09-06T14:31:45.048 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 06 14:31:44 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-iscsi-iscsi-a[59853]: debug Shutdown received 2024-09-06T14:31:55.171 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 06 14:31:54 smithi162 bash[67356]: time="2024-09-06T14:31:54Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-iscsi-iscsi-a in 10 seconds, resorting to SIGKILL" 2024-09-06T14:31:55.171 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 06 14:31:54 smithi162 podman[67356]: 2024-09-06 14:31:54.894745671 +0000 UTC m=+10.126287130 container died 9f3a438e8b62f5e8976b71a5c942ccf58f95de7dc0d4a98469eb1727d382b09b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c86edd48571ab22373283621eb9feebce9df9c291d4164cfc4df1a09493e1f30, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-iscsi-iscsi-a, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-athakkar-testing-2024-09-04-1837-8b38c33, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731) 2024-09-06T14:31:55.548 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 06 14:31:55 smithi162 podman[67356]: 2024-09-06 14:31:55.314499201 +0000 UTC m=+10.546040649 container cleanup 9f3a438e8b62f5e8976b71a5c942ccf58f95de7dc0d4a98469eb1727d382b09b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c86edd48571ab22373283621eb9feebce9df9c291d4164cfc4df1a09493e1f30, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-iscsi-iscsi-a, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, RELEASE=wip-athakkar-testing-2024-09-04-1837-8b38c33, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-09-06T14:31:55.548 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 06 14:31:55 smithi162 bash[67356]: ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-iscsi-iscsi-a 2024-09-06T14:31:55.820 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 06 14:31:55 smithi162 podman[67369]: 2024-09-06 14:31:55.574155454 +0000 UTC m=+0.678528041 container remove 9f3a438e8b62f5e8976b71a5c942ccf58f95de7dc0d4a98469eb1727d382b09b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c86edd48571ab22373283621eb9feebce9df9c291d4164cfc4df1a09493e1f30, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-iscsi-iscsi-a, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-athakkar-testing-2024-09-04-1837-8b38c33) 2024-09-06T14:31:55.820 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 06 14:31:55 smithi162 systemd[1]: ceph-ce32462e-6c5a-11ef-bce4-c7b262605968@iscsi.iscsi.a.service: Main process exited, code=exited, status=137/n/a 2024-09-06T14:31:55.820 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 06 14:31:55 smithi162 systemd[1]: ceph-ce32462e-6c5a-11ef-bce4-c7b262605968@iscsi.iscsi.a.service: Failed with result 'exit-code'. 2024-09-06T14:31:55.820 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 06 14:31:55 smithi162 systemd[1]: Stopped Ceph iscsi.iscsi.a for ce32462e-6c5a-11ef-bce4-c7b262605968. 2024-09-06T14:31:55.820 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 06 14:31:55 smithi162 systemd[1]: ceph-ce32462e-6c5a-11ef-bce4-c7b262605968@iscsi.iscsi.a.service: Consumed 1.995s CPU time. 2024-09-06T14:31:57.513 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:31:57 smithi162 systemd[1]: Stopping Ceph grafana.a for ce32462e-6c5a-11ef-bce4-c7b262605968... 2024-09-06T14:31:57.513 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:31:57 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=server t=2024-09-06T14:31:57.503848283Z level=info msg="Shutdown started" reason="System signal: terminated" 2024-09-06T14:31:57.513 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:31:57 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=ticker t=2024-09-06T14:31:57.504134961Z level=info msg=stopped last_tick=2024-09-06T14:31:50Z 2024-09-06T14:31:57.513 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:31:57 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=tracing t=2024-09-06T14:31:57.50418146Z level=info msg="Closing tracing" 2024-09-06T14:31:57.513 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:31:57 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=grafana-apiserver t=2024-09-06T14:31:57.504216257Z level=info msg="StorageObjectCountTracker pruner is exiting" 2024-09-06T14:31:57.513 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:31:57 smithi162 ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a[62772]: logger=sqlstore.transactions t=2024-09-06T14:31:57.515579841Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-06T14:31:57.767 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:31:57 smithi162 podman[67643]: 2024-09-06 14:31:57.524486848 +0000 UTC m=+0.125701573 container died bfa2d4b5a0ed274434302ea3bcc86b1f5b5a118227dfc17dfef6cb9c1582d7bb (image=quay.io/ceph/grafana:10.4.0, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a, maintainer=Grafana Labs ) 2024-09-06T14:31:58.048 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:31:57 smithi162 podman[67643]: 2024-09-06 14:31:57.837226565 +0000 UTC m=+0.438441281 container cleanup bfa2d4b5a0ed274434302ea3bcc86b1f5b5a118227dfc17dfef6cb9c1582d7bb (image=quay.io/ceph/grafana:10.4.0, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a, maintainer=Grafana Labs ) 2024-09-06T14:31:58.048 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:31:57 smithi162 bash[67643]: ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a 2024-09-06T14:31:58.048 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:31:58 smithi162 podman[67656]: 2024-09-06 14:31:58.004866291 +0000 UTC m=+0.476787633 container remove bfa2d4b5a0ed274434302ea3bcc86b1f5b5a118227dfc17dfef6cb9c1582d7bb (image=quay.io/ceph/grafana:10.4.0, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-grafana-a, maintainer=Grafana Labs ) 2024-09-06T14:31:58.548 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 06 14:31:58 smithi162 systemd[1]: Stopping Ceph node-exporter.b for ce32462e-6c5a-11ef-bce4-c7b262605968... 2024-09-06T14:31:58.548 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:31:58 smithi162 systemd[1]: ceph-ce32462e-6c5a-11ef-bce4-c7b262605968@grafana.a.service: Deactivated successfully. 2024-09-06T14:31:58.548 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:31:58 smithi162 systemd[1]: Stopped Ceph grafana.a for ce32462e-6c5a-11ef-bce4-c7b262605968. 2024-09-06T14:31:58.548 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 06 14:31:58 smithi162 systemd[1]: ceph-ce32462e-6c5a-11ef-bce4-c7b262605968@grafana.a.service: Consumed 6.323s CPU time. 2024-09-06T14:31:58.880 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 06 14:31:58 smithi162 podman[67771]: 2024-09-06 14:31:58.61862358 +0000 UTC m=+0.106522336 container died 07c3262277d3db43782461c7d7ede892cf9ab1a9fc01326948577fbef761c358 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-06T14:31:59.247 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 06 14:31:58 smithi162 podman[67771]: 2024-09-06 14:31:58.974317145 +0000 UTC m=+0.462215901 container cleanup 07c3262277d3db43782461c7d7ede892cf9ab1a9fc01326948577fbef761c358 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-06T14:31:59.247 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 06 14:31:58 smithi162 bash[67771]: ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-b 2024-09-06T14:31:59.545 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 06 14:31:59 smithi162 podman[67783]: 2024-09-06 14:31:59.250571954 +0000 UTC m=+0.632309869 container remove 07c3262277d3db43782461c7d7ede892cf9ab1a9fc01326948577fbef761c358 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-ce32462e-6c5a-11ef-bce4-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-06T14:31:59.545 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 06 14:31:59 smithi162 systemd[1]: ceph-ce32462e-6c5a-11ef-bce4-c7b262605968@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-09-06T14:31:59.545 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 06 14:31:59 smithi162 systemd[1]: ceph-ce32462e-6c5a-11ef-bce4-c7b262605968@node-exporter.b.service: Failed with result 'exit-code'. 2024-09-06T14:31:59.545 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 06 14:31:59 smithi162 systemd[1]: Stopped Ceph node-exporter.b for ce32462e-6c5a-11ef-bce4-c7b262605968. 2024-09-06T14:31:59.545 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 06 14:31:59 smithi162 systemd[1]: ceph-ce32462e-6c5a-11ef-bce4-c7b262605968@node-exporter.b.service: Consumed 2.198s CPU time. 2024-09-06T14:32:00.053 DEBUG:teuthology.orchestra.run.smithi142:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-06T14:32:00.086 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-06T14:32:00.113 INFO:tasks.cephadm:Archiving crash dumps... 2024-09-06T14:32:00.116 DEBUG:teuthology.misc:Transferring archived files from smithi142:/var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/crash to /home/teuthworker/archive/adking-2024-09-06_13:43:09-orch:cephadm-wip-athakkar-testing-2024-09-04-1837-distro-default-smithi/7892315/remote/smithi142/crash 2024-09-06T14:32:00.117 DEBUG:teuthology.orchestra.run.smithi142:> sudo tar c -f - -C /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/crash -- . 2024-09-06T14:32:00.240 INFO:teuthology.orchestra.run.smithi142.stderr:tar: /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/crash: Cannot open: No such file or directory 2024-09-06T14:32:00.240 INFO:teuthology.orchestra.run.smithi142.stderr:tar: Error is not recoverable: exiting now 2024-09-06T14:32:00.242 DEBUG:teuthology.misc:Transferring archived files from smithi162:/var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/crash to /home/teuthworker/archive/adking-2024-09-06_13:43:09-orch:cephadm-wip-athakkar-testing-2024-09-04-1837-distro-default-smithi/7892315/remote/smithi162/crash 2024-09-06T14:32:00.243 DEBUG:teuthology.orchestra.run.smithi162:> sudo tar c -f - -C /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/crash -- . 2024-09-06T14:32:00.292 INFO:teuthology.orchestra.run.smithi162.stderr:tar: /var/lib/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/crash: Cannot open: No such file or directory 2024-09-06T14:32:00.292 INFO:teuthology.orchestra.run.smithi162.stderr:tar: Error is not recoverable: exiting now 2024-09-06T14:32:00.293 INFO:tasks.cephadm:Checking cluster log for badness... 2024-09-06T14:32:00.293 DEBUG:teuthology.orchestra.run.smithi142:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/ceph.log | egrep CEPHADM_ | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'reached quota' | egrep -v 'but it is still running' | egrep -v 'overall HEALTH_' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | egrep -v CEPHADM_STRAY_DAEMON | head -n 1 2024-09-06T14:32:00.326 INFO:tasks.cephadm:Compressing logs... 2024-09-06T14:32:00.326 DEBUG:teuthology.orchestra.run.smithi142:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-06T14:32:00.369 DEBUG:teuthology.orchestra.run.smithi162:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-06T14:32:00.393 INFO:teuthology.orchestra.run.smithi162.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/tcmu-runner.log 2024-09-06T14:32:00.393 INFO:teuthology.orchestra.run.smithi162.stderr:: No such file or directory 2024-09-06T14:32:00.393 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5 --verbose -- /var/log/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/ceph-osd.7.log 2024-09-06T14:32:00.394 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5 --verbose -- /var/log/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/ceph-volume.log 2024-09-06T14:32:00.394 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/tcmu-runner.log: /var/log/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/ceph-osd.7.log: 62.9% -- replaced with /var/log/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/tcmu-runner.log.gz 2024-09-06T14:32:00.394 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5 --verbose -- /var/log/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/ceph-osd.4.log 2024-09-06T14:32:00.395 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/ceph.audit.log 2024-09-06T14:32:00.402 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/ceph.cephadm.log 2024-09-06T14:32:00.402 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/ceph-osd.6.log 2024-09-06T14:32:00.402 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/ceph-osd.5.log 2024-09-06T14:32:00.402 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/ceph-osd.6.log: 79.9% -- replaced with /var/log/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/ceph.cephadm.log.gz 2024-09-06T14:32:00.402 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5 --verbose -- /var/log/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/ceph.log 2024-09-06T14:32:00.402 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/ceph-mgr.x.log 2024-09-06T14:32:00.402 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/ceph-mon.b.log 2024-09-06T14:32:00.402 INFO:teuthology.orchestra.run.smithi162.stderr: 90.2% -- replaced with /var/log/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/ceph.audit.log.gz 2024-09-06T14:32:00.402 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-06T14:32:00.402 INFO:teuthology.orchestra.run.smithi162.stderr: 87.8% -- replaced with /var/log/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/ceph.log.gz 2024-09-06T14:32:00.403 INFO:teuthology.orchestra.run.smithi142.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/ceph-osd.0.log 2024-09-06T14:32:00.403 INFO:teuthology.orchestra.run.smithi142.stderr:: No such file or directory 2024-09-06T14:32:00.403 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/ceph-osd.1.log 2024-09-06T14:32:00.403 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/ceph-volume.log 2024-09-06T14:32:00.403 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/ceph.audit.log 2024-09-06T14:32:00.403 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/ceph-mon.c.log 2024-09-06T14:32:00.404 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/ceph.cephadm.log 2024-09-06T14:32:00.404 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/ceph-mon.a.log 2024-09-06T14:32:00.404 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/ceph-osd.2.log 2024-09-06T14:32:00.404 INFO:teuthology.orchestra.run.smithi142.stderr: 89.3%/var/log/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/ceph-mon.a.log: -- replaced with /var/log/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/ceph.cephadm.log.gz 2024-09-06T14:32:00.404 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/ceph-mgr.y.log 2024-09-06T14:32:00.404 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/ceph.log 2024-09-06T14:32:00.404 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/ceph-mgr.y.log: 93.9% -- replaced with /var/log/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/ceph.audit.log.gz 2024-09-06T14:32:00.404 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/ceph-client.rgw.foo.a.log 2024-09-06T14:32:00.405 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/ceph-mon.b.log: 90.0% -- replaced with /var/log/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/ceph-mgr.x.log.gz 2024-09-06T14:32:00.405 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/ceph.log: 92.9% -- replaced with /var/log/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/ceph.log.gz 2024-09-06T14:32:00.405 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/ceph-osd.3.log 2024-09-06T14:32:00.406 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/cephadm.log: 90.8% -- replaced with /var/log/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/ceph-volume.log.gz 2024-09-06T14:32:00.411 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/ceph-client.rgw.foo.a.log: 91.0% -- replaced with /var/log/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/ceph-volume.log.gz 2024-09-06T14:32:00.412 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-06T14:32:00.420 INFO:teuthology.orchestra.run.smithi162.stderr: 91.0% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-06T14:32:00.425 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/ceph-osd.3.log: /var/log/ceph/cephadm.log: 93.9% -- replaced with /var/log/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/ceph-client.rgw.foo.a.log.gz 2024-09-06T14:32:00.435 INFO:teuthology.orchestra.run.smithi142.stderr: 93.0% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-06T14:32:00.597 INFO:teuthology.orchestra.run.smithi142.stderr: 92.3% -- replaced with /var/log/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/ceph-mon.c.log.gz 2024-09-06T14:32:00.602 INFO:teuthology.orchestra.run.smithi142.stderr: 89.3% -- replaced with /var/log/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/ceph-mgr.y.log.gz 2024-09-06T14:32:00.655 INFO:teuthology.orchestra.run.smithi162.stderr: 92.1% -- replaced with /var/log/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/ceph-mon.b.log.gz 2024-09-06T14:32:00.676 INFO:teuthology.orchestra.run.smithi162.stderr: 92.8% -- replaced with /var/log/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/ceph-osd.6.log.gz 2024-09-06T14:32:00.702 INFO:teuthology.orchestra.run.smithi162.stderr: 93.4% -- replaced with /var/log/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/ceph-osd.4.log.gz 2024-09-06T14:32:00.728 INFO:teuthology.orchestra.run.smithi162.stderr: 93.0% -- replaced with /var/log/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/ceph-osd.7.log.gz 2024-09-06T14:32:00.730 INFO:teuthology.orchestra.run.smithi142.stderr: 92.6% -- replaced with /var/log/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/ceph-osd.2.log.gz 2024-09-06T14:32:00.733 INFO:teuthology.orchestra.run.smithi162.stderr: 92.9% -- replaced with /var/log/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/ceph-osd.5.log.gz 2024-09-06T14:32:00.735 INFO:teuthology.orchestra.run.smithi162.stderr: 2024-09-06T14:32:00.735 INFO:teuthology.orchestra.run.smithi162.stderr:real 0m0.352s 2024-09-06T14:32:00.736 INFO:teuthology.orchestra.run.smithi162.stderr:user 0m1.511s 2024-09-06T14:32:00.736 INFO:teuthology.orchestra.run.smithi162.stderr:sys 0m0.073s 2024-09-06T14:32:00.770 INFO:teuthology.orchestra.run.smithi142.stderr: 93.2% -- replaced with /var/log/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/ceph-osd.1.log.gz 2024-09-06T14:32:00.784 INFO:teuthology.orchestra.run.smithi142.stderr: 93.4% -- replaced with /var/log/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/ceph-osd.3.log.gz 2024-09-06T14:32:00.801 INFO:teuthology.orchestra.run.smithi142.stderr: 93.1% -- replaced with /var/log/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/ceph-osd.0.log.gz 2024-09-06T14:32:00.916 INFO:teuthology.orchestra.run.smithi142.stderr: 90.9% -- replaced with /var/log/ceph/ce32462e-6c5a-11ef-bce4-c7b262605968/ceph-mon.a.log.gz 2024-09-06T14:32:00.918 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-06T14:32:00.918 INFO:teuthology.orchestra.run.smithi142.stderr:real 0m0.535s 2024-09-06T14:32:00.918 INFO:teuthology.orchestra.run.smithi142.stderr:user 0m2.340s 2024-09-06T14:32:00.918 INFO:teuthology.orchestra.run.smithi142.stderr:sys 0m0.108s 2024-09-06T14:32:00.918 INFO:tasks.cephadm:Archiving logs... 2024-09-06T14:32:00.919 DEBUG:teuthology.misc:Transferring archived files from smithi142:/var/log/ceph to /home/teuthworker/archive/adking-2024-09-06_13:43:09-orch:cephadm-wip-athakkar-testing-2024-09-04-1837-distro-default-smithi/7892315/remote/smithi142/log 2024-09-06T14:32:00.919 DEBUG:teuthology.orchestra.run.smithi142:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-06T14:32:01.276 DEBUG:teuthology.misc:Transferring archived files from smithi162:/var/log/ceph to /home/teuthworker/archive/adking-2024-09-06_13:43:09-orch:cephadm-wip-athakkar-testing-2024-09-04-1837-distro-default-smithi/7892315/remote/smithi162/log 2024-09-06T14:32:01.277 DEBUG:teuthology.orchestra.run.smithi162:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-06T14:32:01.513 INFO:tasks.cephadm:Removing cluster... 2024-09-06T14:32:01.513 DEBUG:teuthology.orchestra.run.smithi142:> sudo cephadm rm-cluster --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 --force 2024-09-06T14:32:01.705 INFO:teuthology.orchestra.run.smithi142.stdout:Deleting cluster with fsid: ce32462e-6c5a-11ef-bce4-c7b262605968 2024-09-06T14:32:01.972 DEBUG:teuthology.orchestra.run.smithi162:> sudo cephadm rm-cluster --fsid ce32462e-6c5a-11ef-bce4-c7b262605968 --force 2024-09-06T14:32:02.167 INFO:teuthology.orchestra.run.smithi162.stdout:Deleting cluster with fsid: ce32462e-6c5a-11ef-bce4-c7b262605968 2024-09-06T14:32:02.425 INFO:tasks.cephadm:Teardown complete 2024-09-06T14:32:02.426 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 112, in run_tasks manager.__enter__() File "/usr/lib/python3.10/contextlib.py", line 135, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_8b38c33cecf4b0b7a9b3116eb9089e21e444ac73/qa/tasks/cephadm.py", line 2371, in task healthy(ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_ceph-c_8b38c33cecf4b0b7a9b3116eb9089e21e444ac73/qa/tasks/ceph.py", line 1537, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/git.ceph.com_ceph-c_8b38c33cecf4b0b7a9b3116eb9089e21e444ac73/qa/tasks/ceph_manager.py", line 3299, in wait_until_healthy assert time.time() - start < timeout, \ AssertionError: timeout expired in wait_until_healthy 2024-09-06T14:32:02.642 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=fdcac33f1ddf4d168f68653b80ca2538 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 112, in run_tasks manager.__enter__() File "/usr/lib/python3.10/contextlib.py", line 135, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_8b38c33cecf4b0b7a9b3116eb9089e21e444ac73/qa/tasks/cephadm.py", line 2371, in task healthy(ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_ceph-c_8b38c33cecf4b0b7a9b3116eb9089e21e444ac73/qa/tasks/ceph.py", line 1537, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/git.ceph.com_ceph-c_8b38c33cecf4b0b7a9b3116eb9089e21e444ac73/qa/tasks/ceph_manager.py", line 3299, in wait_until_healthy assert time.time() - start < timeout, \ AssertionError: timeout expired in wait_until_healthy 2024-09-06T14:32:02.644 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-09-06T14:32:02.652 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-06T14:32:02.663 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-06T14:32:02.663 DEBUG:teuthology.orchestra.run.smithi142:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-06T14:32:02.692 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-06T14:32:02.784 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-06T14:32:02.792 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-06T14:32:02.792 DEBUG:teuthology.orchestra.run.smithi142:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-06T14:32:02.794 DEBUG:teuthology.orchestra.run.smithi162:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-06T14:32:02.809 INFO:teuthology.orchestra.run.smithi142.stderr:bash: line 1: ntpq: command not found 2024-09-06T14:32:02.809 INFO:teuthology.orchestra.run.smithi162.stderr:bash: line 1: ntpq: command not found 2024-09-06T14:32:02.814 INFO:teuthology.orchestra.run.smithi142.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-06T14:32:02.814 INFO:teuthology.orchestra.run.smithi142.stdout:=============================================================================== 2024-09-06T14:32:02.814 INFO:teuthology.orchestra.run.smithi142.stdout:^* hv01.front.sepia.ceph.com 3 6 377 17 +804us[ +808us] +/- 40ms 2024-09-06T14:32:02.815 INFO:teuthology.orchestra.run.smithi142.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 21 -1364us[-1360us] +/- 61ms 2024-09-06T14:32:02.815 INFO:teuthology.orchestra.run.smithi142.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 19 +201us[ +206us] +/- 71ms 2024-09-06T14:32:02.815 INFO:teuthology.orchestra.run.smithi142.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-09-06T14:32:02.815 INFO:teuthology.orchestra.run.smithi162.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-06T14:32:02.815 INFO:teuthology.orchestra.run.smithi162.stdout:=============================================================================== 2024-09-06T14:32:02.815 INFO:teuthology.orchestra.run.smithi162.stdout:^* hv01.front.sepia.ceph.com 3 6 377 20 +818us[ +829us] +/- 40ms 2024-09-06T14:32:02.815 INFO:teuthology.orchestra.run.smithi162.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 20 -1310us[-1299us] +/- 61ms 2024-09-06T14:32:02.815 INFO:teuthology.orchestra.run.smithi162.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 22 +241us[ +252us] +/- 71ms 2024-09-06T14:32:02.816 INFO:teuthology.orchestra.run.smithi162.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-09-06T14:32:02.816 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-06T14:32:02.824 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-06T14:32:02.824 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-06T14:32:02.833 DEBUG:teuthology.orchestra.run.smithi142:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-06T14:32:02.931 DEBUG:teuthology.orchestra.run.smithi162:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-06T14:32:02.999 DEBUG:teuthology.orchestra.run.smithi142:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-06T14:32:03.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-06T14:32:03.028 DEBUG:teuthology.orchestra.run.smithi162:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-06T14:32:03.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-06T14:32:03.056 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-06T14:32:03.065 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-06T14:32:03.074 INFO:teuthology.task.internal:Duration was 1218.315964 seconds 2024-09-06T14:32:03.074 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-06T14:32:03.082 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-06T14:32:03.082 DEBUG:teuthology.orchestra.run.smithi142:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-06T14:32:03.085 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-06T14:32:03.123 INFO:teuthology.orchestra.run.smithi142.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-06T14:32:03.134 INFO:teuthology.orchestra.run.smithi162.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-06T14:32:03.497 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-06T14:32:03.497 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi142.front.sepia.ceph.com 2024-09-06T14:32:03.497 DEBUG:teuthology.orchestra.run.smithi142:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-06T14:32:03.558 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi162.front.sepia.ceph.com 2024-09-06T14:32:03.559 DEBUG:teuthology.orchestra.run.smithi162:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-06T14:32:03.597 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-06T14:32:03.597 DEBUG:teuthology.orchestra.run.smithi142:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-06T14:32:03.601 DEBUG:teuthology.orchestra.run.smithi162:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-06T14:32:03.663 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-06T14:32:03.663 DEBUG:teuthology.orchestra.run.smithi142:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-06T14:32:03.669 DEBUG:teuthology.orchestra.run.smithi162:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-06T14:32:04.082 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-06T14:32:04.091 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-06T14:32:04.091 DEBUG:teuthology.orchestra.run.smithi142:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-06T14:32:04.116 DEBUG:teuthology.orchestra.run.smithi162:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-06T14:32:04.142 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-06T14:32:04.151 DEBUG:teuthology.orchestra.run.smithi142:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-06T14:32:04.158 DEBUG:teuthology.orchestra.run.smithi162:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-06T14:32:04.179 INFO:teuthology.orchestra.run.smithi142.stdout:kernel.core_pattern = core 2024-09-06T14:32:04.205 INFO:teuthology.orchestra.run.smithi162.stdout:kernel.core_pattern = core 2024-09-06T14:32:04.222 DEBUG:teuthology.orchestra.run.smithi142:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-06T14:32:04.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-06T14:32:04.252 DEBUG:teuthology.orchestra.run.smithi162:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-06T14:32:04.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-06T14:32:04.277 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-06T14:32:04.286 INFO:teuthology.task.internal:Transferring archived files... 2024-09-06T14:32:04.286 DEBUG:teuthology.misc:Transferring archived files from smithi142:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/adking-2024-09-06_13:43:09-orch:cephadm-wip-athakkar-testing-2024-09-04-1837-distro-default-smithi/7892315/remote/smithi142 2024-09-06T14:32:04.287 DEBUG:teuthology.orchestra.run.smithi142:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-06T14:32:04.333 DEBUG:teuthology.misc:Transferring archived files from smithi162:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/adking-2024-09-06_13:43:09-orch:cephadm-wip-athakkar-testing-2024-09-04-1837-distro-default-smithi/7892315/remote/smithi162 2024-09-06T14:32:04.333 DEBUG:teuthology.orchestra.run.smithi162:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-06T14:32:04.368 INFO:teuthology.task.internal:Removing archive directory... 2024-09-06T14:32:04.368 DEBUG:teuthology.orchestra.run.smithi142:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-06T14:32:04.371 DEBUG:teuthology.orchestra.run.smithi162:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-06T14:32:04.418 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-06T14:32:04.429 INFO:teuthology.task.internal:Not uploading archives. 2024-09-06T14:32:04.429 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-06T14:32:04.438 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-06T14:32:04.439 DEBUG:teuthology.orchestra.run.smithi142:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-06T14:32:04.441 DEBUG:teuthology.orchestra.run.smithi162:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-06T14:32:04.453 INFO:teuthology.orchestra.run.smithi142.stdout: 262217 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 6 14:32 /home/ubuntu/cephtest 2024-09-06T14:32:04.473 INFO:teuthology.orchestra.run.smithi162.stdout: 262217 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 6 14:32 /home/ubuntu/cephtest 2024-09-06T14:32:04.475 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-06T14:32:04.484 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-06T14:32:04.503 INFO:teuthology.run:Summary data: description: orch:cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rados_api_tests} duration: 1218.315964460373 failure_reason: timeout expired in wait_until_healthy flavor: default owner: scheduled_adking@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=fdcac33f1ddf4d168f68653b80ca2538 status: fail success: false 2024-09-06T14:32:04.504 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-06T14:32:04.581 INFO:teuthology.run:FAIL